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Key findings 


Community college students often lack an academic plan to guide their choice 
of coursework and achieve their education goals, in part because counseling 
departments typically lack the capacity to advise all students. This randomized 
controlled trial tested the impact of guaranteed access to either a group workshop 
or a one-on-one academic counseling session to help students prepare an academic 
plan, along with reminders to attend the sessions. Both interventions increased 
academic plan completion rates by more than 20 percentage points over completion 
rates for a control group that received neither guaranteed access to a counseling 
session nor ongoing electronic “nudges” to attend. Exploratory evidence suggests 
that workshop counseling is as effective as one-on-one counseling in getting 
students to complete the academic planning process. Workshop counseling was the 
most cost-effective counseling option based on completion rates of academic plans. 
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Summary 


Getting more college students to prepare a semester-by-semester academic plan is widely 
considered a promising strategy for improving persistently low completion rates at com- 
munity colleges (Scott-Clayton, 2011). Nationwide, only about 35 percent of students who 
enroll in community college earn a credential within six years (Radford, Berkner, Whee- 
less, & Shepherd, 2010). In California only one in four students enrolled in community 
college earns a credential or transfers to a four-year college within six years of first enroll- 
ing (Moore, Shulock, Ceja, & Lang, 2007). Statistics such as these prompted the Califor- 
nia State Legislature, with the strong support of the Chancellor’s Office of the California 
Community Colleges, to pass the Student Success Act in 2012. The act mandates that all 
community college students complete an academic plan early in their college career, but 
provides no guidance to colleges on how to operationalize the mandate. The challenges of 
doing so are formidable. Given that the median national community college ratio of stu- 
dents to counselors is 441 to 1 (Robbins, 2013), most counseling departments lack enough 
counselors to work with each student individually to develop a plan. Usually, only the 
more motivated students endure the long waits to see a counselor (Scott-Clayton, 2011; 
Venezia, Bracco, & Nodine, 2010). 

To overcome these challenges, the South Orange County Community College District 
developed a technology-based approach to bring academic planning to scale while ensur- 
ing that all plans receive counselor input. For years students have had access to an online 
academic planning tool called My Academic Plan (MAP), which was designed and devel- 
oped by the district. In collaboration with Regional Educational Laboratory West, the dis- 
trict used the MAP tool to test an intervention that guaranteed access to one of two types 
of counseling sessions (group workshops or one-on-one counseling), combined with tar- 
geted “nudging” to encourage students to attend the counseling session and complete an 
academic plan — that is, submitting a counselor-approved academic plan created using the 
MAP tool. Nudges were delivered through Sherpa, the district’s web-based platform that 
disseminates information to students on college courses, deadlines, academic programs, 
and other support to help them navigate college. The district also developed an integrated 
data system to coordinate MAP, Sherpa, and SARS, the scheduling software program, 
which together constitute the MAP system. 

This report presents the results of a randomized controlled trial that tested whether guar- 
anteeing access to either a group counseling workshop with standardized content or a 
regular one-on-one counseling session, combined with nudging to get students to attend 
the session, increases the likelihood that students would complete an academic plan com- 
pared with students in a control group that received no guaranteed access to counseling 
and no personalized electronic reminders. The study also tested the extent to which stu- 
dents assigned to either intervention group are more or less likely than students in the 
control group to schedule and attend a counseling session and register for coursework the 
following semester. 

In September 2014, 1,763 first-time students at Saddleback College were randomly assigned 
to one of three groups: 1,085 students were assigned to the workshop group, 193 students 
to the one-on-one counseling group, and 485 students to the control group. Students in 
the control group received neither the guaranteed access to counseling nor the ongoing 
nudging. Instead, those students could seek help in completing a plan by scheduling a 


counseling appointment or waiting in line for an appointment like any other Saddleback 
student. The study yielded four key findings: 

• The enhanced MAP system was implemented as intended by the South Orange 
County Community College District and Saddleback College. Most notably, the 
nudges were successfully delivered, and the counseling workshops were conducted 
according to the schedule and plan. 

• Both the workshop and one-on-one counseling interventions increased the per- 
centage of students who scheduled and attended counseling appointments and 
who completed an academic plan by more than 20 percentage points compared 
with the control group. 

• Neither the workshop nor the one-on-one counseling interventions appeared to 
affect student enrollment in the following semester. 

• Exploratory evidence suggests that the workshop intervention, with nudges, was 
less costly and performed as well as the one-on-one counseling intervention, as 
measured by the percentage of students who successfully scheduled and attended 
counseling appointments and completed an academic plan. 
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Why this study? 


Nationwide, only about 35 percent of students who enroll in community college com- 
plete a certificate, associate’s degree, or bachelor’s degree from any institution within six 
years (Radford et al., 2010). In California only one in four students enrolled in community 
college earns a credential or transfers to a four-year college within six years of first enroll- 
ing (Moore et al, 2007). Statistics such as these prompted the California State Legislature, 
with the strong support of the Chancellor’s Office of the California Community Colleges, 
to pass the Student Success Act in 2012 (SB 1456). The act requires all first-time commu- 
nity college students, starting with the fall 2014 cohort, to take a placement test to assess 
their readiness for college-level courses, to receive orientation, and to prepare a semes- 
ter-by-semester academic plan listing their education goals, major, and courses needed to 
fulfill their major requirements. While the act mandated that all community college stu- 
dents have a comprehensive academic plan, it provides no guidance to colleges or how to 
operationalize the mandate. 

Academic planning is widely considered to be a promising strategy for improving persistent- 
ly low completion rates at community colleges (Scott-Clayton, 2011). Intuitively, having 
an academic plan early in one’s college career makes sense. If students make informed 
decisions about their career goals, their major, and the courses necessary to satisfy the 
requirements for their major, they should be more likely to make steadier progress toward 
achieving their goals. Students consistently agree with this idea when surveyed (Booth 
et al., 2013; Center for Community College Student Engagement, 2012; Matus-Grossman, 
Gooden, Wavelet, Diaz, & Seupersad, 2002). 

At the time that the Student Success Act was passed, most California community col- 
leges required students to take placement tests and attend orientation programs but did 
not require students to complete an academic plan. Some colleges encouraged academic 
planning, especially through special classes or programs, but participation was low. The act 
required that, as of fall 2015, all students submit an academic plan before earning 15 semes- 
ter units or before completing their third semester. Students who did not submit an academ- 
ic plan are barred from registering for classes the following semester. Given many colleges’ 
severely limited counseling capacity, colleges were concerned about their ability to work 
with hundreds or even thousands of students each semester to prepare an academic plan. 

The South Orange County Community College District was motivated by the mandate 
of the Student Success Act to advance its longstanding goal to scale academic planning. 
Home to two community colleges serving nearly 40,000 students, the district was well 
positioned to advance an academic planning agenda (see appendix A for more information 
about the district and its student population). Several years earlier the district’s informa- 
tion technology department had created two technology-based planning tools: an online 
academic planning tool called My Academic Plan (MAP) and a recommendation and 
personalization engine designed to enhance student success called Sherpa. The district 
had been using Sherpa for years to “nudge” students via email messages, text messages, 
notices on students’ MySite web portals, and robocalls to meet campus deadlines and take 
advantage of campus services. As the district began planning to use Sherpa to guide more 
students through the MAP process, the counseling departments at the district’s two com- 
munity colleges (Irvine Valley and Saddleback) were in the process of determining how 
to deal with potentially thousands of students seeking guidance for and approval of their 
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plans. The counseling department also decided to require that every student’s academic 
plan be reviewed and approved by a counselor, a requirement that goes beyond those out- 
lined in the Student Success Act. 

The operational challenges of providing all students with an approved academic plan, 
based on one-on-one guidance from a counselor — the traditional approach to helping 
students create an academic plan — are formidable, given the national median commu- 
nity college ratio of students to counselors of 441 to 1 (Robbins, 2013). Because access 
to counselors is limited, usually only the more motivated students endure the long waits 
to see a counselor (Scott-Clayton, 2011; Venezia, Bracco, & Nodine, 2010). The South 
Orange County Community College District knew that its system to equip students with 
a counselor-approved academic plan was not adequate for satisfying the act’s mandates. For 
example, in fall 2012 less than 10 percent of students opened the district’s MAP tool. 

Prompted by the 2012 act to learn more about what it would take to increase the per- 
centage of students who completed an academic plan, the district collaborated with the 
California Community College Alliance at Regional Educational Laboratory (REL) West 
to launch an experiment in fall 2014. The effort began one year before registration bars 
were scheduled to go into effect for students in the fall 2014 cohort of new students who 
did not submit a counselor-approved academic plan. The district believed that the best way 
to achieve this goal was to run counselor-led group sessions, rather than require students to 
schedule an appointment or wait in line for a one-on-one counselor meeting. Together, the 
district and one of its colleges, Saddleback College, devised an intervention, referred to in 
this report as the “MAP system,” to increase the percentage of students with a completed 
academic plan (see box 1 for a detailed description of the system’s four key components). 


Box 1. The four components of the My Academic Plan system 

A completed academic plan is one created using the MAP tool that has been reviewed and 
approved by a counselor. The academic plan lists the courses a student plans to take, semes- 
ter by semester, including those needed to satisfy the requirements of a major and to attain 
the student’s goals (for example, an associate’s degree or transfer to another college), as well 
as any needed developmental education classes, general education courses, and electives. 

For this study of the MAP system, students were randomly assigned to three groups: two 
intervention groups, with “nudging” and guaranteed group or individual counselling, and one 
control group, which received neither nudges nor guaranteed access to counseling. 

1. Targeted nudging. Students randomly assigned to an intervention group received target- 
ed and personalized “nudging” (designed by college staff with help from the study team) 
through emails, text messages, notices on students’ MySite web portals, and robocalls, 
to urge them to start and complete the MAP process (see table A1 in appendix A). These 
nudges were delivered by Sherpa, a recommendation and personalization engine. 

2. Guaranteed counseling services. Intervention group students were offered a guaranteed 
counseling appointment, held for up to two weeks following each nudge, either in a two- 
hour group counseling workshop, which followed a counselor-designed curriculum, or in 
a one-on-one counseling session conducted as a regular counseling session. Guaran- 
teed counseling was organized to maximize the number of intervention group students 
who would take advantage of it and make an appointment, while not negatively affecting 

(continued) 
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Box 1. The four components of the My Academic Plan system (continued) 


access to counselors by control group students and students not in the study. To ensure 
that the study did not lengthen student wait times to make appointments and see a coun- 
selor, the college’s counseling department implemented a number of strategies, including 
making greater use of adjunct counseling faculty who also work as part-time counselors. 

3. Access to an online academic planning tool. All students (in intervention and control 
groups as well as those who were not in the study) had access to the MAP tool. The tool 
guides students through an online process to select courses needed to satisfy their major 
and meet their education goals. The tool is linked to online resources including course 
catalogs, requirements for transfer to four-year institutions, and other useful information 
to help students and their counselors construct a comprehensive, semester-by-semester 
plan. The tool has been available to all students at the college for several years and was 
made available to the control group as well as to the two intervention groups. 

4. Integrated data system. An integrated data system, developed by the South Orange County 
Community College District and Saddleback College, was used to coordinate nudges, track 
student responses, schedule workshops and one-on-one sessions, and track attendance 
and academic plan completion. The system incorporated MAP, Sherpa, and SARS, a sched- 
uling software that was adapted for the MAP system (see figure D1 in appendix D for more 
details about the system). This system was developed specifically for the intervention. 


What the study examined 


The study used a randomized controlled trial to assess how two interventions affected 
completion rates for student academic plans. An academic plan was considered completed 
if it was created using the MAP tool and was reviewed and approved by a counselor. 

All eligible students (those who were reported as new to the district and who had not yet 
completed an academic plan) in the freshman cohort of fall 2014 at Saddleback College 
were randomly assigned to one of three study groups: one of two intervention groups or 
a control group (see box 2 for an overview of the study data and random assignment of 
students and appendixes A, B, and C for a detailed discussion of the study methods and 
data). Both intervention groups received guaranteed access to a counselor and a set of 
personalized nudges over the course of the semester to encourage them to take advantage 
of the counseling guarantee and complete an academic plan. Students in the intervention 
groups were randomly assigned to either the workshop group or the one-on-one group. The 
control group received only an initial nudge, an email in which the student was encour- 
aged to make an appointment or walk in to see a counselor to complete an academic plan 
(table 1). This represented business as usual at the college. 


An academic plan 
was considered 
completed if it 
was created using 
the MAP tool and 
was reviewed 
and approved 
by a counselor 


The study assessed whether nudging students to attend and guaranteeing them access 
to either a counseling workshop or a one-on-one counseling session increased academic 
plan completion rates compared with business as usual (control group). Students in the 
control group, like other Saddleback students, received only one nudge email and were not 
guaranteed a counseling session. The study also compared rates of students making and 
keeping counseling appointments and registering for classes the following semester for the 
intervention and control groups. 
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Box 2. Data sources and study sample 


Data sources 

To assess fidelity of implementation, the study relied mostly on qualitative data, gathered 
in focus groups and interviews, to measure how the My Academic Plan (MAP) system was 
implemented and to capture the different perceptions and attitudes of students, counselors, 
and college administrators. The implementation study also took advantage of the college’s 
student survey, which students (including a small number of nonstudy participants) completed 
before leaving the counseling workshops. To estimate impacts on the outcomes studied, the 
study relied on data obtained from college administrative records, including demographic and 
enrollment data, attendance in workshops and one-on-one counseling sessions, and academic 
plan completion data. The cost and cost-effectiveness analyses used financial data from Sad- 
dleback College. (For a more detailed description of qualitative data collection methods and 
data used to measure outcomes, see appendix B.) 

Study sample 

Random assignment ensures that the characteristics of students in each of the three research 
groups are unlikely to be different at the start of the study. By comparing the behavior of 
students who received customized nudges with the behavior of students who did not, and 
by comparing academic plan completion rates of students who were assigned to each of the 
three study groups, it is possible to determine whether the intervention caused the observed 
differences in outcomes among groups. 

A random sample was selected in the third week of September 2014, a few weeks after 
classes had begun. All students who met the eligibility criteria (including being a first-time 
student at Saddleback and not having previously completed an academic plan) were automat- 
ically enrolled in the study but were given an opportunity to opt out prior to random assign- 
ment (see appendix A for a discussion of the eligibility criteria). After accounting for students 
who opted out of the study and for ineligible students, the analysis sample included 1,763 
students: 1,085 students were randomly assigned to the workshop group, 193 students 
to the one-on-one group, and 485 students to the control group. The numbers of students 
assigned to the workshop and one-on-one groups were informed by two factors: the capacity 
of the college’s counseling department to dedicate counselors to delivering MAP services, 
and requirements in the research design to enable detection of statistically significant differ- 
ences between the intervention groups and the control group. (See appendix B for a detailed 
explanation of how statistical power was calculated.) The college decided that offering 6 
group sessions and 20 one-on-one sessions per week for 10 weeks was optimal, given the 
number of counselors available. These parameters determined the maximum number of stu- 
dents who were assigned to these two study groups, with the remainder assigned to the 
control group. 

Random assignment resulted in three study groups that look similar in gender, age, race, 
ethnicity, reasons for attending college, and prior education level completed (see tables Cl and 
C2 in appendix C). Under randomization, it is expected that around 5 percent of all comparisons 
of baseline characteristics are statistically significant at the p < .05 level (.05 * 78 = 4). There- 
fore, observing 4 or 5 statistically significant differences is neither unusual nor cause for 
concern. Characteristics of the students in the sample generally reflect those of students in 
the college as a whole (see table A1 in appendix A), and there were few statistically signifi- 
cant differences among students randomly assigned to the three groups. This means that any 
postintervention differences in the outcomes are likely due to the intervention itself. 

(continued) 
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Box 2. Data sources and study sample (continued) 

A small number of students assigned to one study group ended up working on their academic 
plans as part of another study group. The analysis in this report is an intent-to-treat analysis, so 
these students’ outcomes were analyzed as part of the group to which they were assigned rather 
than the group in which they participated. Ten control group students accessed a workshop during 
the study and two completed an academic plan as part of a course. One control group student 
accessed the one-on-one counseling session designated for the one-on-one group students. 


Table 1. My Academic Plan intervention components and how they vary by study 
group, 2014/15 


Workshop group 

One on one group 

Control group 1 

Targeted 

nudging 

Initial nudge and up to 10 additional 
nudges, ceasing after student has 
completed the My Academic Plan 
(MAP) process 

Initial nudge and up to 

10 additional nudges, 
ceasing after student has 
made an appointment 
and completed the MAP 

process 

Initial email nudge only 

Guaranteed 

access to 

counseling 

services 

Guaranteed appointments can be 

made for reserved slots within a 

two-week period through a link sent 
via email or phone call; workshop 
curriculum walks students through 
the process of completing an 
academic plan; held in computer 
lab, two hours long, one to two 
counselors present, capped at 19 

students 

Guaranteed appointments 

can be made for reserved 

slots within a two-week 

period through a link sent 
via email or phone call; 

one-on-one session with 

counselor; about one hour 
long 

Not guaranteed; students 
can make an appointment 
for the following week 
or walk in; one-on-one 
session with counselor; 

a small number of 

students can receive MAP 

counseling as part of a 
college course 

Source: Authors' compilation. 


Whether students assigned to the workshop group completed academic plans at the same 
rate as students assigned to the one-on-one group was also explored. Although the answer 
to this question is important to the district, the study was not powered to definitively deter- 
mine whether group counseling was as effective as one-on-one counseling. Thus, this evi- 
dence should be interpreted as exploratory rather than confirmatory. (Appendix B gives a 
more detailed explanation of the difference between confirmatory and exploratory analyses.) 

Finally, the study compared the costs of the three groups in terms of student attendance in 
counseling sessions and academic plans completed. 

The study was designed to answer questions about whether the MAP system increased the 
number of students who started and completed the MAP process. It also looked at how 
the system was implemented, how counselors and students reacted to it, and the relative 
cost-effectiveness of the two interventions and the control condition. 

Implementation questions: 

• Was the intervention implemented as intended by the South Orange County 
Community College District and Saddleback College? 

• How did college administrators, counselors, and students perceive the MAP 
system, the counseling workshops, and the factors affecting their implementation? 
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Impact questions: 

• Does assignment to the workshop group increase the academic plan completion 
rate, compared with the control group? 

• Does assignment to the one-on-one group increase the academic plan completion 
rate, compared with the control group? 

• Does assignment to the workshop group decrease the academic plan completion 
rate, compared with assignment to the one-on-one group? 

• Do the percentages of students who make and attend counseling appointments 
differ across the three groups? 

• Do the percentages of students who persist into the subsequent semester differ 
across the three groups? 


The study used 
a randomized 
controlled trial 
to assess how 
two interventions 
affected 
completion 
rates for student 
academic plans 


Cost-effectiveness question: 

• How did the average cost per student differ across the three groups with respect to 
MAP, counseling session, and completion rates? 


What the study found 


This section presents three types of findings corresponding to the three types of research 
questions: implementation findings, impact findings, and cost and cost-effectiveness findings. 

Implementation findings: The MAP system was implemented as planned 

The South Orange County Community College District and Saddleback College spent 
nearly a year planning for the launch of the MAP system, assisted by REL West. Following 
the system’s launch in September 2014, REL West assisted the college in collecting infor- 
mation to describe the implementation process, including qualitative information from 
focus groups and interviews. This section of the report describes implementation of the key 
components of the MAP system based on these data. (See figure A1 in appendix A for a 
logic model of the MAP system depicting how it was designed.) 

Targeted nudges were generally delivered on schedule without major technological prob- 
lems. The district’s information technology department, with technical assistance from 
the REL West study team, created distinct profiles of students based on their study group 
assignments. These profiles triggered Sherpa, a district-developed recommendation and 
personalization engine, to deliver corresponding nudges to students. The South Orange 
County Community College District’s information technology department also connected 
Sherpa with SARS — an off-the-shelf appointment system adopted by the district and inte- 
grated with its online education planning tool — to personalize the content of the nudges 
and track appointments made by students. The two systems worked in tandem to identify 
intervention group students who had not scheduled a counseling appointment or who had 
scheduled an appointment but had not kept it. Sherpa sent subsequent nudges only to stu- 
dents who had not yet attended a workshop or a one-on-one counseling session. 

The district delivered up to 11 nudges to each student assigned to the workshop and one- 
on-one groups throughout the semester (see table A2 in appendix A). All students across 
the three study groups received an initial email nudge informing them about the MAP 
tool, the consequences if they did not complete an academic plan by the time they earned 
15 semester units, and the opportunity to opt out of the study if they wished. Initial email 


6 


nudges to the two intervention groups also encouraged students to schedule a counseling 
appointment using a link included in the email. After this initial nudge only students in the 
two intervention groups continued to receive nudges (up to 10 additional nudges through 
January 12, 2015). Five types of nudges were sent over the semester: emails, MySite To-Do 
List items (see figure D3 in appendix D), robocalls, emails with a short video, and text 
messages (see table A2 in appendix A). The study was not designed to assess which type of 
nudge worked best; however, according to anecdotal evidence from counseling staff, robo- 
calls elicited the strongest response from students. As one staff member noted, “Within an 
hour of sending out the phone-call nudge, we got 75 phone calls [to set up appointments].” 

All study and nonstudy students were exposed to campuswide MAP promotion, including 
coverage in student orientation and during the matriculation process, posters and flyers 
around the campus, classroom announcements by counselors, and a page on the Saddle- 
back College website to inform students about the need to complete an academic plan (see 
figure A1 in appendix A). 

Students in intervention groups could make guaranteed appointments over a two week 
period. Students in the workshop and one-on-one groups could make a MAP counsel- 
ing appointment in two ways: through a direct link included in their electronic nudges 
or by calling or visiting the counseling department. Slots in a MAP counseling session 
were guaranteed, and appointments could be scheduled for any offered time in the follow- 
ing two weeks. Students in the control group could only follow the standard practice of 
making a MAP counseling appointment by calling or visiting the counseling department, 
and they could only reserve a MAP counseling session for the following week (rather than 
the following two weeks). In addition, control group students were not guaranteed a slot 
in a session; they had to compete with other students for limited MAP counseling slots, as 
students always had, and slots for a given week were often filled by Monday afternoons of 
the previous week. Control group students could also try to make a walk-in appointment 
to see a counselor, but during busy times this could involve long waits. 

Counseling workshops provided students with the structure and staffing to support 
their completion of an academic plan. The workshops were structured to ensure that stu- 
dents understood why and how to create an academic plan, could work on and complete 
their academic plan during the workshop, had access to counselors, and could leave the 
workshop as soon as their academic plan was reviewed and approved by the counselor (box 
A1 in appendix A shows the counselor-designed curriculum). To accommodate students’ 
busy schedules, workshops were scheduled four days a week and were run regardless of how 
many students attended. 

Counselors conducted 57 two-hour workshops during the fall semester, between September 
29 and December 18, 2014. Workshops were held at the college’s counseling department, in 
a computer lab set up for this purpose. The lab was equipped with 19 desktop computers, as 
well as a projector and a screen. 

During the semester 612 (56.4 percent) of the students assigned to the workshop group 
made appointments to attend a workshop, and 523 (48.2 percent) of the students assigned 
to the workshop group attended a workshop. Workshop attendance ranged from 1 to 14 stu- 
dents, with an average of 6 students per workshop. Most workshops were attended by more 
than six students, but sparse attendance at the first few workshops lowered the average. 1 


Students in the 
workshop and 
one-on-one 
groups could 
make a counseling 
appointment 
through a direct 
link included in 
their electronic 
nudges or by 
calling or visiting 
the counseling 
department. Slots 
in a counseling 
session were 
guaranteed 
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Administrators and counselors attributed early low attendance to inadequate publicity 
about the MAP tool early in the semester. Also, the first nudge was erroneously sent after 
the first workshop was scheduled, resulting in no students showing up for the first workshop. 

Nearly all the workshops (95 percent) were facilitated by one or two counselors from a 
pool of nine part-time counselors. (The effect of this spread of counselors is discussed in 
appendix B.) The college’s counseling department assigned its part-time counselors to lead 
the workshops because their schedules allowed them to be slotted into a set of scheduled 
workshops more easily than full-time counselors. To standardize the quality of counsel- 
ing across the two intervention groups, the counseling department arranged for training 
early in the semester by a lead counselor in order to familiarize counselors with the MAP 
tool and the structure and process for the workshops. The director of counseling services 
reported that the majority of counselors received six to eight hours of training. A typical 
counseling workshop is described in box 3. 


Box 3. A typical counseling workshop 

Students file into the workshop computer lab as a counselor stands next to a computer at 
the front of the room. The room’s projector flickers on, and students begin to quiet down 
behind individual computer screens at their seats. The counselor explains that, for the next 30 
minutes or so, she will describe My Academic Plan (MAP), its purpose, and how to complete 
one. Students introduce themselves one by one, naming their major and whether they have 
heard of MAP. The counselor defines the MAP tool and displays MySite through the projector. 
Students log in to their MySite web portals, and a few begin to work on their academic plan, 
either for the first time or, if they had started one previously, picking up where they left off. An 
aide roves around the room to help latecomers log in. 

The counselor explains the types of courses that need to be included in an academic plan: 
general-education requirements, electives, prerequisites, and major courses. For students who 
plan on transferring, she suggests reviewing University of California and California State Uni- 
versity requirements, displaying on the screen the website ASSIST.org, showing the courses 
that satisfy the Intersegmental General Education Transfer Curriculum and the California State 
University General Education requirements for transfer. 

The counselor then walks through the steps for creating an academic plan on her screen, 
including choosing a major and courses that fulfill requirements needed for their major or to 
transfer to another college. A few hands go up, and the counselor answers student questions. 
About 35 minutes into the workshop, the counselor’s presentation is over, and students begin 
to work in earnest, sometimes speaking to the student next to them or peeking at each other’s 
screens. The counselor walks around the room answering questions and approving students’ 
academic plans. This continues as students complete their academic plans one by one. An 
hour into the workshop about half the students have had their academic plans approved and 
have left the room. Students waiting for their academic plans to be approved form a queue, 
and a second counselor joins the first counselor in approving students’ academic plans and 
answering questions. This continues for about 15 minutes, until the two counselors have fin- 
ished approving each student’s academic plan. At that point, the first counselor checks the 
box next to the name of each student with an approved academic plan in the college data 
system that communicates with Sherpa to stop sending reminders to the student. 

Note: This description of a typical workshop is a composite based on several workshop observations. Some 
students left a session without an approved academic plan for various reasons including needing more time to 
reflect on their education or career goals before completing the plan. 


Counselors 
conducted 57 two- 
hour workshops 
during the fall 
semester, and 
during the semester 
612 (56.4 percent) 
of the students 
assigned to the 
workshop group 
made appointments 
to attend a 
workshop, and 
5 23 (48.2 percent) 
of the students 
assigned to 
the workshop 
group attended 
a workshop 
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One-on-one counseling sessions, conducted in individual counseling offices, provided 
students with the information needed to complete their academic plan. Counselors 
conducted one-on-one counseling sessions with 72 (37.3 percent) of the students eligible 
for that intervention. These sessions occurred between September 25 and December 17, 
2014. One-on-one sessions were scheduled to last an hour. Eleven full-time counselors led 
61 percent of the sessions, while six part-time counselors led the remaining 39 percent. A 
typical session is described in box 4. 

One-on-one counseling sessions operated as one-on-one sessions have always operated at the 
college, and they differed from the workshops in important ways. As one counselor explained, 

As always, when they’re coming to see us individually, the first task is to find out 
why they’re there, what they need, what they want, and then work with that, 
whatever that is. So that’s very different than doing an orientation in how to put 
[a] MAP together. And we’re putting it together often with them. They’re not 
sitting at the computer. Sometimes they have one done, and we’re reviewing it, if 
that’s what they need to have done. We’re reviewing it with them. Or we’re creat- 
ing it with them. So that’s quite different than a workshop. 


Box 4. A typical one-on-one counseling session 

A counselor walks into her office with a student from the waiting area. The counselor sits in 
front of her computer screen, and the student sits next to the desk. The counselor introduces 
herself and asks the student about his academic plan and if he has heard about My Academic 
Plan (MAP) yet. The student expresses interest in electrical engineering and in transferring to a 
University of California school, but he’s not completely sure what an academic plan is. When the 
counselor asks him how sure he is about his goals on a scale of 1-10, the student states “a 7.” 

The counselor then shares a paper that details transfer requirements for different Univer- 
sity of California schools and explains the types of course and credit requirements needed and 
typical course options for each. The counselor asks the student what classes he liked in high 
school, explaining that this might give him a good idea of what courses to take to fulfill trans- 
fer requirements. The counselor then looks at her computer screen to confirm the student’s 
registration date and suggests that the student set up an appointment with her to register for 
classes on that date. 

Fifteen minutes into the session, the counselor pulls up the MAP tool on her computer 
screen, and the student inches his chair closer. She notices that he had already started cre- 
ating an academic plan on his own. She walks through the student’s academic plan with him, 
asking which of the course choices for each requirement sound interesting. As he answers, 
the counselor inserts courses into the MAP tool accordingly. She explains that these choices 
are not set in stone, and she asks the student if the process is making sense to him. For 
every class selected, the counselor describes details (such as professors, course popular- 
ity, and content). The counselor continues inserting courses into the MAP tool to create a 
semester-by-semester plan according to the student’s input. This takes about 35 minutes. 
The student reviews the chosen courses, and the counselor signs and prints out a copy of the 
academic plan for the student. With 10 minutes left in the hour-long session, she makes the 
student a folder with materials and reminds him to come to see her during registration. 

Note: Some students left the session without an approved academic plan for various reasons including need- 
ing more time to reflect on their education or career goals before completing the plan. 
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Students were generally satisfied with the workshops. According to a college-adminis- 
tered survey completed by approximately 163 students immediately after working on their 
academic plans in the workshops but before leaving their computers, about 90 percent of 
survey respondents agreed that they were satisfied with the workshop, and about 84 percent 
agreed that the workshop met their academic planning needs (see table C4 in appendix 
C). 2 In open-ended responses students reported that the workshops helped them to “lay 
out the classes [they] needed to take” and to “plan for the future” and that the counsel- 
ors were “very helpful” (see figure Cl). Some students liked working together and hearing 
the advice given to other students, and many valued the individual attention that they 
received from the counselors in the workshops. Suggested improvements included adding 
more counselors and allowing for more individual attention from counselors (see figure 
C3). As one student put it, overall, the workshops were “easy, simple, and necessary.” One 
student reported that the workshop exceeded expectations, explaining, “I expected just to 
have my original MAP [academic plan] confirmed and locked in, but it ended up being a 
lot more helpful than I thought it was going to be.” 


Students in the 
workshops liked 
working together 
and hearing the 
advice given to 
other students, 
and many valued 
the individual 
attention that 
they received from 
the counselors 


Despite initial concerns of some counselors that the group format of the workshops 
might diminish the usefulness of academic plans, students reported feeling confident in 
their completed academic plans. During the planning and early implementation phases 
of the MAP system, some counselors expressed concern that they would not be able to 
provide students with enough individualized attention to help them complete a useful and 
meaningful academic plan. 3 Four of the eight counselors in the focus group explained that 
students arrived at the workshops “all over the map” and “at different developmental stages 
and [with different] plan[s] .” As a result, these counselors felt “pulled in many directions” 
and worried that their assistance was therefore less effective than it would have been in 
one-on-one sessions. As one counselor stated, “It’s when they’re all in different places, dif- 
ferent majors, undecided, it’s difficult to feel that confidence when you look at their plan, 
because you haven’t done that one-on-one time.” 


Notwithstanding some counselors’ concerns, all workshop students who participated in 
focus groups reported feeling confident about their academic plans. As one workshop 
student explained, “I am pretty confident. I feel a lot more relaxed, though, because I know 
what classes I’ll be taking each semester. I’ll have a plan, and it’s really helped me.” In addi- 
tion, according to the college-administered survey, about 89 percent of workshop students 
agreed that they intended to use their academic plans to reach their academic goals at 
Saddleback (see table C4 in appendix C). 


Impact findings: A higher percentage of the intervention group students than the control group 
students completed academic plans 

Students assigned to the workshop or one-on-one counseling with nudging and guaran- 
teed access to counseling were significantly more likely to attend a counseling session and 
to complete an academic plan than students in the control group. There was no signifi- 
cant difference in session attendance or academic plan completion between the workshop 
and one-on-one counseling groups. Nudging and guaranteed access to counseling clearly 
increased the completion rate but failed to ensure that all students completed an academic 
plan: a majority of students in all three groups did not complete an academic plan. 
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Students assigned to either intervention group who received nudging and a guaranteed 
counseling appointment were significantly more likely to schedule and attend appoint - 
ments than were students in the control group. About 56 percent of students assigned 
to the workshop group scheduled a workshop appointment, and 48 percent attended one, 
while about 33 percent of control group students scheduled a counseling appointment, and 
30 percent attended one (table 2). The differences between the outcomes for the work- 
shop and control groups are statistically significant. Similarly, about 59 percent of students 
assigned to the one-on-one group scheduled a counseling appointment, an estimated dif- 
ference of about 26 percentage points over the control group. About 50 percent of the one- 
on-one group attended their appointments, an estimated difference of about 20 percentage 
points over the control group. These differences are large and statistically significant. 

Students who were assigned to either intervention group and who received nudging and 
a guaranteed counseling appointment were significantly more likely to complete an aca- 
demic plan than were students in the control group. Over the five months of the study, 
about 38 percent of students randomly assigned to the workshop group and 42 percent of 
the students randomly assigned to the one-on-one counseling group completed an aca- 
demic plan, compared with about 19 percent of students in the control group (see table 2). 
This resulted in a 20 percentage point difference between workshop group students and 
control group students and a 23 percentage point difference between one-on-one-group 
students and control group students. Both differences are statistically significant. 


There was no 
significant 
difference in 
session attendance 
or academic 
plan completion 
between the 
workshop and 
one-on-one 
counseling groups 


Nevertheless, a majority of students — 62 percent in the workshop group, 58 percent in the 
one-on-one group, and 81 percent in the control group — failed to complete an academic 
plan. While nudging and guaranteed access to counseling sessions improved completion 
rates, additional intervention is needed to ensure that all students engage in academic 
planning. 


Table 2. Differences in percentages of students who scheduled and attended My Academic Plan 
counseling sessions, completed academic plans, and enrolled the following spring, by study group, 
2014/15 


Outcome measure 

Workshop 

group 

(percent) 

One-on-one 

group 

(percent) 

Control 

group 

(percent) 

Difference 
between 
workshop and 
control groups 
(percentage 
points) 

Difference 
between one- 
on-one and 
control groups 
(percentage 
points) 

Difference 
between 
workshop and 
one-on-one groups 
(percentage 
points ) 3 

Student scheduled 
counseling appointment 

56.4 

58.5 

32.6 

23 8*** 

26 0*** 

-2.1 

Student attended 
counseling appointment 

48.2 

50.3 

30.1 

18.1*** 

20 2*** 

-2.1 

Student completed 
academic plan 

38.2 

42.0 

18.8 

19.5*** 

23 2 *** 

-3.7 

Student enrolled in spring 
2015 (persistence) 

77.8 

77.2 

79.8 

-2.0 

-2.6 

0.6 


*** Significant at p < .001. 

Note: The total sample size was 1,763. There were 1,085 students in the workshop group, 193 students in the one-on-one group, and 
485 students in the control group. Rounding may cause slight discrepancies in differences. Statistical tests were conducted using 
t-test statistics generated by ordinary least squares regressions. See appendix B for model details. 

a. Describes findings related to the exploratory research questions. Values should be interpreted with caution. The lower sample size 
for the comparisons between the one-on-one counseling group and the workshop group means that causal inference is not possible. 

Source: Authors' calculations using 2015 administrative data from the South Orange County Community College District. 
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There were no statistically significant differences between the two intervention groups 
in the percentage of students who scheduled or attended counseling appointments or 
completed an academic plan. Students assigned to the workshop group scheduled and 
attended counseling appointments at nearly the same rate as students assigned to the one- 
on-one group (see table 2). The same is true for completion rates. This finding suggests 
that the workshops and the one-on-one counseling sessions have similar effects on the 
outcomes analyzed in this study, but limitations in the study design suggest caution in 
directly comparing results for the two intervention groups (see section on Limitations of 
the study). 4 

Many intervention group students failed to take the first step of scheduling an appoint - 
ment. Being assigned to the workshop or one-on-one counseling groups increased the 
percentage of students who scheduled and attended appointments. However, more than 
40 percent of students assigned to the workshop or one-on-one counseling groups did not 
schedule a counseling appointment. About 85 percent of students from the workshop 
group and about 86 percent of one-on-one group followed through on their appointments. 
While the percentage of control group students who made a counseling appointment 
(33 percent) and who attended a counseling appointment (30 percent) was much smaller, 
some 92 percent of the control group students who did schedule an appointment also 
attended the appointment. 


While nudging and 
guaranteed access 
to counseling 
sessions improved 
completion 
rates, additional 
intervention 
is needed to 
ensure that all 
students engage in 
academic planning 


Neither intervention appeared to affect enrollment in the following semester. About the 
same percentages of students in each study group enrolled in the following semester (spring 
2015): about 78 percent of students assigned to the workshop group, about 77 percent of 
students assigned to the one-on-one group, and about 80 percent of students assigned to 
the control group (see table 2). No enrollment differences between any of the groups were 
statistically significant. 


Cost-effectiveness findings: The workshop group was the most cost effective of the two 
interventions, based on cost per academic plan completed 

The per student cost of counseling was highest for the one-on-one counseling group. 
The workshop group and the control group had similar costs per student. The average 
per-student cost was $27 for the workshop group, $46 for the one-on-one group, and $24 for 
the control group (table 3). See table C5 in appendix C for additional details about how 
costs presented in table 3 were calculated. 

The costs in table 3 assume that counseling is delivered by a mix of full-time and part-time 
counselors. If all sessions were taught by full-time (more expensive) counselors, the cost per 
student would increase for each group: the workshop group cost would increase from $27 to 
$34, the one-on-one group cost increase from $46 to $53, and the control group cost would 
increase from $24 to $28 (see table C5 in appendix C). 

The Ivor ks hop group was the most cost-effective option when the cost of each coun- 
seling approach is compared to the corresponding outcomes. Cost-per-outcome values 
are estimated by dividing the cost of each group by the total number of desired outcomes 
(counseling sessions attended and academic plans completed) in that group. These out- 
comes were selected for the cost analysis as they were the primary outcomes in the impact 
analysis. Comparing these values shows how cost effective each approach was at producing 
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Table 3. Costs and cost-effectiveness of workshop and one-on-one counseling 
groups for My Academic Plan compared with the control group, 2014/15 


Outcome 

Workshop 

group 3 

One-on-one 

group 

Control 

group 

Cost per study student (dollars) 

27 

46 

24 

Percentage of students attending a counseling session 

48.2 

50.3 

30.1 

Cost per counseling session attended (dollars) 

55 

91 

79 

Percentage of students completing an academic plan 

38.2 

42.0 

18.8 

Cost per academic plan completed (dollars) 

70 

109 

126 

Number of observations 

1,085 

193 

485 


Note: All costs are in 2014 dollars. For detailed notes about how costs were calculated, see table C5 in 
appendix C. 

a. The estimated cost per student is based on an average session size of 7.95 students. The cost per student 
would fall if the average workshop size increased. 

Source: Cost estimates are calculated from program expenditure data reported by Saddleback College and 
salary information for full-time and part-time counselors reported by the California State Controller's Office. 
Impacts on attending counseling and completing an academic plan are estimated using 2015 administrative 
data from the South Orange County Community College District. 


desired outcomes compared with other approaches. The workshop group had a lower cost 
per student outcome than the one-on-one group and a similar cost per student outcome to 
the control group. Additionally, the workshop group produced outcomes that were similar 
to those of the one-on-one group but better than the control group. As a result, the work- 
shop group is the most cost-effective counseling group (lowest cost per desired outcome). 

The workshop group was the most cost-effective with respect to students attending coun- 
seling sessions. The cost per counseling session attended was lowest (most cost-effective) 
for the workshop group and highest (least cost-effective) for the one-on-one group. Specif- 
ically, the cost per counseling session attended for the workshop group was $55, compared 
with $91 for the one-on-one group and $79 for the control group (see the third row of table 
3). An alternative cost per counseling session attended was estimated that assumes all 
counseling was provided by full-time counselors rather than by the actual mix of full-time 
and part-time counselors (see table C5 in appendix C). This assumption does not change 
the results, as the cost per counseling session attended remains lowest for the workshop 
group and highest for the one-on-one group. 

Similarly, the workshop group was the most cost effective with respect to the students 
completing academic plans. The cost per academic plan completed was lowest (most cost 
effective) for the workshop group and highest (least cost effective) for the control group. 
Specifically, the cost per academic plan completed for the workshop group was $70 (see the 
fifth row of table 4). In comparison, the costs per academic plan completed were $109 for 
the one-on-one group and $126 for the control group. Cost per academic plan completed 
values are also estimated for an alternative scenario in which all groups receive counseling 
from full-time counselors only. The results are unaffected by this assumption. 

Implications of the study findings 


The workshop 
group had a lower 
cost per student 
outcome than the 
one-on-one group 
and a similar 
cost per student 
outcome to the 
control group 
and produced 
outcomes that 
were similar to 
those of the one- 
on-one group but 
better than the 
control group 


This randomized controlled trial offers evidence that targeted nudging to encourage 
students to attend either workshop or one-on-one counseling sessions and guaranteed 
access to these counseling sessions significantly increases the likelihood that students will 
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complete an academic plan, compared with students who did not have access to these 
resources. The study also suggests that students who receive the interventions are more 
likely than control group students to schedule and attend a counseling appointment. From 
a fiscal perspective the study suggests that holding counseling workshops is a promising, 
cost-effective strategy to ensure that greater numbers of students complete academic plans. 
Counseling workshops substantially increased the number of students who completed an 
academic plan at a lower cost per desired outcome than one-on-one counseling or “busi- 
ness as usual.” The study found that the components of the MAP system (targeted nudging, 
guaranteed counseling services, online academic planning tool, and the integrated data 
system) were implemented, for the most part, as intended and were operational for the 
duration of the study. This allowed for successful scaling of counseling support for student 
academic planning. It also resulted in making the process of completing an academic plan 
for both counselors and students “easy, simple, and necessary,” as described by a student. 

The district’s experience suggests that colleges seeking scalable, sustainable, efficient, and 
effective methods to encourage students to complete an academic plan could use an online 
system that nudges students and guarantees them counseling appointments to achieve 
similar results, as long as they have access to the required technology to take full advan- 
tage of limited counseling capacity. 5 

It is important to note that although the nudging and guaranteed counseling components 
of the MAP system significantly increased the number and percentage of students who 
completed an academic plan, they did not work for all students. About half the students in 
the intervention groups took the first step of making an appointment; the other half did 
not, even after receiving up to 11 nudges encouraging them to do so during the semester 
and being offered guaranteed counseling services. This study shows that while nudging 
and guaranteed access to group and individual counseling clearly helped get more students 
to initiate and complete the academic planning process, additional support is needed to 
get all students to complete an academic plan. California’s decision to bar students from 
registering for classes if they fail to complete an academic plan by the time they earn 15 
credits will boost — and already has boosted — the percentage of students who complete 
an academic plan. In states where such a policy does not exist, group counseling, enabled 
by technology, may be one of the most effective tools colleges can use to scale academic 
planning. 

Student success is at the heart of why the South Orange County Community College Dis- 
trict invested the resources and effort to create and evaluate a new approach to academic 
planning. The MAP system was designed to harness the power of technology, use existing 
counseling capacity more efficiently, and facilitate increased access to counseling services 
for larger numbers of students. As one district administrator put it: 

In California we have a very real challenge ahead of us that ultimately could 
impact the success of millions of students. Very few dispute the benefit of stu- 
dents creating and following an academic plan; however, we don’t have enough 
counselors to adequately guide students in this complex process. The answer is to 
use intelligent technology that leverages the scarce and valuable resource of coun- 
selors and broadens their impact beyond the traditional one-on-one counseling 
sessions. 


This study offers 
evidence that 
targeted nudging 
to encourage 
students to attend 
either workshop 
or one-on-one 
counseling sessions 
and guaranteed 
access to these 
counseling sessions 
significantly 
increases the 
likelihood that 
students will 
complete an 
academic plan 
and suggests that 
holding counseling 
workshops is a 
promising, cost- 
effective strategy 
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Limitations of the study 


This study has six main limitations. First, it lacked sufficient power to provide more than 
suggestive evidence that counseling workshops are as effective as one-on-one counseling 
in getting students to complete an academic plan. While researchers observed no statisti- 
cally significant differences between the percentages of students who completed a plan in 
these two study groups, the limited sample size may have masked meaningful differences. 

Second, the study was not designed to test the relative importance of nudging and the 
guaranteed access to a counseling session. Understanding the role that each of these com- 
ponents played in changing student behavior would be important information for colleges 
designing future student support programs. 

Third, because of time constraints, the study did not track student persistence for more 
than one semester. Although there were no significant group differences in persistence 
(enrollment in the semester after the intervention), differences could emerge over time. It 
is possible that the differences would increase as students earn 15 credits, after which time 
their registration is blocked unless they have an approved academic plan. The majority 
of the students in the sample had not earned 15 credits by the time they registered for 
the following semester, meaning that students were able to register without a completed 
academic plan. 

Fourth, the study was not designed to compare the quality of the academic plans across 
the three study groups, as measured by, for example, how useful they are for students or 
whether students adhere to the plans as they progress toward their goals. This information 
could be important given the concerns expressed by some counselors about the quality of 
the academic plans completed in the workshop sessions as compared with those prepared 
in traditional one-on-one sessions. Quality measures could have varied by intervention 
group, particularly as they relate to student outcome measures. This could be important 
information for colleges to have as they make decisions about counseling programs for 
education planning. 

Fifth, because the college disproportionately assigned part-time staff to teach the work- 
shops, the impact estimates may be biased due in part to the characteristics of part-time 
counselors, which may have differed from full-time counselors in ways that mattered for 
the outcomes. For example, tenured full-time staff may be less motivated than adjuncts to 
deliver counseling in a group setting or rely on technology to guide their interactions with 
students. 

Finally, and perhaps most important, the study was not designed to answer the critical 
question of whether the process of developing an academic plan improves students’ chances 
of successfully attaining academic goals and transitioning to a career. As an increasing 
number of colleges require students to develop education plans, the time has come for 
rigorous studies that address the impact of academic planning on academic and career 
outcomes. 
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Appendix A. Study background and intervention characteristics 


This appendix includes the descriptions, tables, and figures that supplement the infor- 
mation about the study in the report narrative. Included are descriptions of the college’s 
setting and student population; a My Academic Plan (MAP) system logic model and illus- 
trations of the student flow through the MAP system; and the counseling workshop cur- 
riculum, the content and format of the nudges sent to students, and a description of the 
comparison conditions. 

Study setting 

The study took place in the South Orange County Community College District in south- 
ern California. The district serves a suburban community approximately one hour’s drive 
from downtown Los Angeles. The district is home to two colleges that are relatively auton- 
omous: each has its own president and administrative staff and a somewhat distinct edu- 
cation culture. Saddleback College, the college that participated in the study, serves the 
city of Mission Viejo; Irvine Valley College serves the city of Irvine. About 33,000 students 
were enrolled in the district in fall 2014, with approximately 20,000 of them attending 
Saddleback and 13,000 attending Irvine Valley. At Saddleback, students can earn an asso- 
ciate of arts degree or certificate in more than 190 academic and career program areas. 
According to the Integrated Postsecondary Education Data System data from the National 
Center for Education Statistics, during the 2014/15 academic year, 54 percent of Saddle- 
back students were White, 9 percent were Asian, 25 percent were Hispanic/Latino, and 
2 percent were Black (table Al). Sixteen percent of district students receive Pell grants, 
which provide need-based grants to low-income college students. 


Table Al. Characteristics of students at Saddleback College, fall 2014 

Student characteristic 

Number or percent 

Enrollment 

Total undergraduate enrollment 

20,007 

Receiving Pell grant (percent) 

16 

Gender (percent) 

Male 

46 

Female 

54 

Race/ethnicity (percent) 

Asian or Pacific Islander 

9 

Black 

2 

Hispanic 

25 

White 

54 

Other 

10 

Age (percent) 

Younger than age 18 

2 

Ages 18-24 

62 

Ages 25-64 

34 

Age 65 and older 

1 


Note: Percentages may not sum to 100 because of rounding. 

Source: Authors' calculations using 2014 data from the National Center for Education Statistics Integrated 
Postsecondary Education Data System. 
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Saddleback had 14 full-time counselors and 23 part-time counselors at the time of the 
study. The full-time counselors divide their time between counseling and teaching. During 
the study part-time counselors led most of the workshops, while full-time counselors pro- 
vided most of the counseling for students in the one-on-one and control groups. The 
college provided training to all counselors on how to use the MAP tool and how to advise 
students in the group setting, which may have helped to mitigate variations in counselors’ 
experience and familiarity with the MAP tool. 

This study used a random assignment design with three conditions. Students were ran- 
domly assigned to one of three groups: one of the two intervention groups (the workshop 
group or the one-on-one group, both of which involved the nudging and guaranteed coun- 
seling services of the MAP system) or the control group, which did not receive an inter- 
vention beyond what was available to all students. 

The My Academic Plan system 

The development and testing of the MAP system was a collaborative effort by the South 
Orange County Community College District, Saddleback Community College, and 
Regional Educational Laboratory (REL) West. The intervention was designed to increase 
the number of incoming students who complete a counselor-approved semester-by-semester 
plan listing the courses they need to achieve their education goal, whether it is a certifi- 
cate, an associate’s degree, or transfer to a four-year institution. With such a plan, it was 
hoped, students would be more likely to graduate from college within a reasonable time 
and earn a credential that would help them transition to the workforce or to a four-year 
institution. 

The MAP system consists of four key components, listed in box 1 in the main report 
and the logic model in figure Al, which presents the resources, components, outputs, and 
short- and long-term outcomes of the MAP system. Some elements of the components 
were in place in the district before the study began, some were new, and others improved 
on existing strategies. What is new about the MAP system is that it brought together all of 
the components in a coherent, seamless system that made creating an academic plan and 
having it reviewed and approved by a counselor easier for both counselors and students. 

Intervention condition 

Students in the two intervention groups were assigned to nudges and guaranteed access to 
one of two counseling methods. The two intervention groups received essentially identical 
treatment, except that one involved random assignment to group counseling (the work- 
shop group), and the other involved random assignment to individual counseling sessions 
with a counselor (the one-on-one group). 

The ideal student flow through the MAP system for workshop and one-on-one student 
groups at Saddleback College is illustrated in figure A2. The workshop curriculum devel- 
oped by Saddleback College counselors is shown in box Al. The curriculum was available 
for internal use only and is not available elsewhere. The nudges sent to students during the 
course of the study are described in table Al. 
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Figure Al. The My Academic Plan (MAP) system logic model, Saddleback College, 
2014/15 


Resources 


Components 


Outputs 


Outcomes 

Short-term 


Long-term 


Technology 


Counselors 


Leadership 

support 



Targeted and 
personalized 
nudges 


Guaranteed spot 
in a one-on-one 
session or 
MAP workshop 


Up to 11 nudges 
issued throughout 
semester, ceasing 
as students respond 


1,140 MAP 
workshop slots 


200 one-on-one 
rrf session slots PF 


Increased 
scheduling of 
appointments for 
academic plan 
counseling sessions 


Increased 
attendance at 
academic plan 
counseling sessions 



Increased 
adherence to the 
academic plan 


Increased 
persistence in 
following semesters 


Access to the 
MAP tool 


Continuously 
improved MAP 
tool and 
tracking system 


Increased 
academic plan 
completion rate 


Increased credits 
attempted 
and earned 


Support 

staff 


An integrated 
online nudging, 
scheduling, and 
tracking system 


Six to eight hours 
of counselor 
training offered 


Increased 
persistence in the 
following semester 


Accelerated and/or 
increased credential 
attainment 


♦ ♦ ♦ ♦ ' ± 

Contextual factors: Number and quality of counselors, information diffusion about 
and need for academic plans, student characteristics, state and district policies 


Note: Boxes with dotted lines represent outcomes not measured in this study. 
Source: Authors' compilation. 


Comparison condition 

The control group was free to access any counseling service or other supports available 
to all students at the college to complete an academic plan, including the MAP tool and 
counseling support. However, these students did not receive the intensive nudging or 
the guaranteed access to counseling that were provided to the intervention groups. As 
part of the study, control group students received only a single email nudge explaining 
how they could opt out of the study and advising them to complete an academic plan. 
The completion process for control group students is illustrated in figure A3. Although 
10 control group members were registered as having attended a workshop designated for 
workshop-group students (and are therefore considered to be “crossovers”), this was not 
permissible according to the study protocols. The small number of crossovers had no sig- 
nificant effect on the findings. 

The existence of the study did not appear to affect the control condition. That is, control 
group students had the same access to the online MAP tool and the same access to coun- 
selors that existed before the study. The counseling department ensured such access by 
calling in a sufficient number of part-time counselors to meet the additional MAP-related 
demand on the department that the study generated. According to interviews conducted 
with administrators in the counseling department, the wait time to make an appointment 
or drop in to see a counselor for the control group students was not lengthened due to the 
study. 
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Figure A2. Student flow for My Academic Plan workshop and one-on-one 
intervention groups, Saddleback College, 2014/15 



Note: Thick arrows indicate the ideal path for workshop and one-on-one counseling students. 
Source: Authors' compilation. 


Box Al. Curriculum for My Academic Plan workshops developed by Saddleback 
College counselors, 2014/15 

Introduction to the My Academic Plan tool (2 minutes) 

Explain to students that the My Academic Plan (MAP) tool is a computerized academic plan- 
ning program and the purpose of the workshop is to develop a comprehensive academic 
plan. Remind students that they will need to have a completed comprehensive academic plan 
approved by a counselor by the time they reach 15 units of credit. 

MySite/My Academic Plan (3 minutes) 

• Show students how to access the MAP tool through MySite and how to choose a goal of a 
Certificate, associate degree, or transfer. 

• Explain that for the associate degree and transfer goals, students will need to complete 
general education and major preparation requirements as presented on MAP. 

• Discuss choosing a major and the use of ASSIST (an online student-transfer information 
system that shows how course credits earned at one California college or university can 
be applied when transferred to another) in identifying major preparation requirements (ask 
students who are undecided to choose a Liberal Studies major). 

(continued) 
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Box Al. Curriculum for My Academic Plan workshops developed by Saddleback 

College counselors, 2014/15 (continued) 

General education (5 minutes) 

• Hand outgeneral education patterns flat-sheets (Native pattern, California State University 
General Education, University of California or California State University Intersegmental 
General Education Transfer Curriculum) and explain that each student who plans on receiv- 
ing an associate degree or transferring will need to follow one of these general education 
patterns and will be asked to select one to complete an academic plan. 

• Discuss math and English placement scores and explain the numbering system used to 
identify if a course counts for the associate degree or transfers to the California State 
University system or the University of California system. 

• Discuss clearing of prerequisites. 

Major (ASSIST or Saddleback College Catalog) and electives (3 minutes) 

• Show students how to select a college and a major using the MAP tool and have students 
select a Liberal Studies major if undecided regarding choice of major. 

• Explain the use of electives to reach the 60 units needed for an associate degree or to 
transfer and how to select electives using the MAP tool. 

Example of a completed comprehensive education plan — page 2 of the academic plan from 

the My Academic Plan tool (2 minutes) 

• Let students know that this is the page you want them to show you when they have a com- 
pleted plan. Point out course sequences in math and English. 

Semester-to-semester plan (100 minutes) 

• Have students access the MAP tool through MySite and develop a comprehensive educa- 
tion plan. 

• Assist students with accessing the MAP tool, setting an education goal, and choosing 
classes. 

Review and approve academic plan and check off as completed 

• Use the online list to check off names of students whose academic plans are complet- 
ed (reviewed and approved by a counselor) to meet the comprehensive education plan 
requirement. 

Closing comments — remind students about 

• Counseling appointments. 

• Walk-in counseling. 

• Applied psychology classes. 

• Advanced Placement scores. 


Table A2. Reminders (nudges) sent to Saddleback College students in study sample, 2014/15 


Nudge Group Format Date Content 


1 Control group Email 9/25/14 Hi {FirstName}, 

Welcome to Saddleback! Do you know that as a new student, you are now 
required to complete a comprehensive academic plan? And do you know that if 
you do not have a counselor review this plan by the time you earn 15 units you 
may not be able to register for classes? 

Here at Saddleback, we have a great online tool called "My Academic Plan” 
(MAP) students use to create a plan. 

Why is having a MAP a great idea? 

1. A MAP can keep you on track because it spells out the courses you need to 
meet all requirements for your major and general education. 

2. A MAP is not set in stone: you can make changes anytime! 

3. All students must have a counselor-reviewed MAP or they may not be able to 
register for classes! 

You are among a group of students who have been selected to participate in a 
special study the college is conducting this Fall to learn the best ways to help 
all students at Saddleback complete a MAP. As part of this study, Saddleback 
will share your student records data with a research project through REL West. 
The data includes personal identifiers, such as name and student ID. In the 
future, researchers may contact you to ask questions about your experiences 
completing your educational plan. Information that REL West and Saddleback 
collect for the study will be kept confidential and will be used for research 
purposes only. Participation in the study will not affect your standing in college 
or your access to any normal services. If you have questions about the study, 
or you don’t want to be contacted or your data to be shared, please contact 
Saddleback Counseling at (949) 582-4570. 


(continued) 
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Table A2. Reminders (nudges) sent to Saddleback College students in study sample, 2014/15 (continued) 


Nudge Group Format Date Content 


1 Workshop Email 9/25/14 Hi {FirstName}, 

Welcome to Saddleback! Do you know that as a new Saddleback student, you 
are now required to complete a comprehensive academic plan? And do you know 
that if you do not have a counselor review this plan by the time you earn 15 units 
you may not be able to register for classes? 

Here at Saddleback, we have a great online tool called "My Academic Plan” 

(MAP) students use to create a plan. 

You have been selected to attend a special 2-hour-long workshop in which a 
Saddleback counselor will walk you through the steps to complete a MAP. No 
waiting in line! To take advantage of this opportunity and to reserve a space 
in one of these workshops click here: http://www.saddleback.edu/esars/ 
mdrcwrkshp or call (949) 582-4570. 

Why is having a MAP a great idea? 

1. A MAP can keep you on track because it spells out the courses you need to 
meet all requirements for your major and general education. 

2. A MAP is not set in stone: you can make changes anytime! 

3. All students must have a counselor-reviewed MAP or they may not be able to 
register for classes! 

You are among a group of students who have been selected to participate in a 
special study the college is conducting this Fall to learn the best ways to help 
all students at Saddleback complete a MAP. As part of this study, Saddleback 
will share your student records data with a research project through REL West. 
The data includes personal identifiers, such as name and student ID. In the 
future, researchers may contact you to ask questions about your experiences 
completing your educational plan. Information that REL West and Saddleback 
collect for the study will be kept confidential and will be used for research 
purposes only. Participation in the study will not affect your standing in college 
or your access to any normal services. If you have questions about the study, 
or you don’t want to be contacted or your data to be shared, please contact 
Saddleback Counseling at (949) 582-4570. 


(continued) 
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Table A2. Reminders (nudges) sent to Saddleback College students in study sample, 2014/15 (continued) 


Nudge Group 


One-on-One 


Email 


9/25/14 Hi {FirstName}, 


Welcome to Saddleback! Do you know that as a new Saddleback student, you 
are now required to complete a comprehensive academic plan? And do you know 
that if you do not have a counselor review this plan by the time you earn 15 units 
you may not be able to register for classes? 

Here at Saddleback, we have a great online tool called "My Academic Plan” 

(MAP) students use to create a plan. 

You have been selected to meet one-on-one with a counselor for 60 minutes 
to complete your MAP in which a Saddleback counselor will walk you through 
the steps to complete a MAP. No waiting in line! To take advantage of this 
opportunity and to schedule a counseling appointment click here: http://www. 
saddleback.edu/esars/mdrcindv or call (949) 582-4570. 

Why is having a MAP a great idea? 

1. A MAP can keep you on track because it spells out the courses you need to 
meet all requirements for your major and general education. 

2. A MAP is not set in stone: you can make changes anytime! 

3. All students must have a counselor-reviewed MAP or they may not be able to 
register for classes! 

You are among a group of students who have been selected to participate in a 
special study the college is conducting this Fall to learn the best ways to help 
all students at Saddleback complete a MAP. As part of this study, Saddleback 
will share your student records data with a research project through REL West. 
The data includes personal identifiers, such as name and student ID. In the 
future, researchers may contact you to ask questions about your experiences 
completing your educational plan. Information that REL West and Saddleback 
collect for the study will be kept confidential and will be used for research 
purposes only. Participation in the study will not affect your standing in college 
or your access to any normal services. If you have questions about the study, 
or you don’t want to be contacted or your data to be shared, please contact 
Saddleback Counseling at (949) 582-4570. 


2 

Workshop 

MySite to-do 

list item 

9/25/14 

Sign up to see a counselor to complete your MAP. Click here http://www. 
saddleback.edu/esars/mdrcwrkshp or call (949) 582-4570. 

2 

One-on-one 

MySite to-do 

list item 

9/25/14 

Sign up to see a counselor to complete your MAP. Click here http://www. 
saddleback.edu/esars/mdrcindv or call (949) 582-4570. 

3 

Workshop 

Email with 

Flipagram 

10/6/14 

This flipagram was made just for you — http://youtu.be/JCXp8ebKVB4. Get your 
MAP completed and approved by a counselor. Click here http://www.saddleback. 
edu/esars/mdrcwrkshp or call (949) 582-4570. 

3 

One-on-one 

Email with 
Flipagram 

10/6/14 

This flipagram was made just for you — http://youtu.be/JCXp8ebKVB4. Get your 
MAP completed and approved by a counselor. Click here http://www.saddleback. 
edu/esars/mdrcindv or call (949) 582-4570. 

4 

Workshop 

Robocall 

10/17/14 

This is the Saddleback College Counseling Department. We noticed that you haven't 
made a workshop appointment yet. This puts you at risk for losing your priority 
registration status. Luckily, it’s not too late to schedule a meeting with a counselor. 
Call us at (949) 582-4570 to schedule your workshop appointment today. 

4 

One-on-one 

Robocall 

10/17/14 

This is the Saddleback College Counseling Department. We noticed that you 
haven’t made a MAP appointment yet. This puts you at risk for losing your priority 
registration status. Luckily, it’s not too late to schedule a meeting with a counselor. 
Call us at (949) 582-4570 to schedule your counseling appointment today. 

5 

Workshop 

Email 

11/3/14 

Spring registration will begin soon! Don't lose your priority registration. Complete 
your MAP today and have it approved by a counselor! To reserve your space in 
a workshop click here: http://www.saddleback.edu/esars/mdrcwrkshp or call 
(949) 582-4570 to get started. 


(continued) 
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Table A2. Reminders (nudges) sent to Saddleback College students in study sample, 2014/15 (continued) 

1 Nudge 

Group 

Format 

Date 

Content 

5 

One-on-one 

Email 

11/3/14 

Spring registration will begin soon! Don't lose your priority registration! Complete 
your MAP today and have it approved by a counselor! To make an appointment 
with a counselor click here: http://www.saddleback.edu/esars/mdrcindv or call 
(949) 582-4570 to get started. 

6 

One-on-one 

and workshop 

Text 

message 

11/3/14 

Saddleback College Reminder — Complete your MAP! Don't lose your priority 
registration! Call (949) 582-4570 to see a counselor. 

7 

Workshop 

Email 

11/10/14 

You haven’t made a MAP appointment yet. Don't lose your priority registration 
status. Click here to attend a workshop and have your academic plan approved by a 
counselor: http://www.saddleback.edu/esars/mdrcwrkshp or call (949) 582-4570. 

7 

One-on-one 

Email 

11/10/14 

You haven't made a MAP appointment yet. Don’t lose your priority registration 
status. Click here to schedule an individual appointment with a counselor to 
have your academic plan approved: http://www.saddleback.edu/esars/mdrcindv 
or call (949) 582-4570. 

8 

Workshop 

Email 

12/2/14 

Spring registration has begun and completing a MAP will help keep you on 
track to meeting your educational goal. You can still attend a workshop session 
and have a counselor approve your academic plan! Click here: http://www. 
saddleback.edu/esars/mdrcwrkshp or call (949) 582-4570 to reserve a spot. 

8 

One-on-one 

Email 

12/2/14 

Spring registration has begun and completing a MAP will help keep you on track 
to meeting your educational goal. You can still attend a MAP counseling session 
and have a counselor approve your academic plan! Click here: http://www. 
saddleback.edu/esars/mdrcindv or call (949) 582-4570 to reserve a spot. 

9 

Workshop and 

one-on-one 

Text 

message 

12/11/14 

Saddleback College Reminder — Next week is the last chance to get your MAP 
approved by a counselor for fall. Call (949) 582-4570. 

10 

Workshop 

Email 

1/7/15 

Happy New Year! Get the New Year off to a great start! Complete your MAP today 
and have it approved by a counselor. To reserve your space in a workshop click 
here: http://www.saddleback.edu/esars/mdrcwrkshp or call (949) 582-4570 to 
get started. 

10 

One-on-one 

Email 

1/7/15 

Happy New Year! Get the New Year off to a great start! Complete your MAP today 
and have it approved by a counselor. To make an appointment with a counselor 
click here: http://www.saddleback.edu/esars/mdrcindv or call (949) 582-4570 
to get started. 

11 

Workshop and 

one-on-one 

Text 

message 

1/12/15 

Saddleback College Notice — Maintain your priority registration status; reserve a 
seat at a MAP counseling session. Call (949) 582-4570. 

Source: Authors' compilation. 
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Figure A3. Business as usual: My Academic Plan completion process for control 
group students, 2014/15 



Note: Thick arrows indicate the ideal path for control group students. 
Source: Authors' compilation. 
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Appendix B. Study data sources, design and analysis 


This appendix describes the data sources, study participants, random assignment, defini- 
tions of outcome measures, analytic approach, statistical adjustments, treatment of missing 
data, and qualitative data collected and analyzed. 

Quantitative data sources 

A total of 1,839 students were randomly assigned into the study. Random assignment was 
conducted at the student level. Due to college preferences and capacity considerations, 
assignment to the one-on-one group was capped at 200 students and assignment to the 
workshop group was capped at 1,140. This left a maximum of 499 slots for the control 
group. 6 Within the 1,839 students who were randomly assigned, 24 students opted out of 
the study, and 52 students were found to have completed an academic plan prior to random 
assignment. These students were therefore ineligible for the study and were removed from 
the analytic sample, leaving 1,763 students. 

Data were collected for all 1,815 students who did not opt out of the study, including the 
52 students who were found to be ineligible because they had completed an academic plan 
prior to random assignment. The ineligible students were dropped from the sample before 
the impact models were run. The district’s database provided data on counseling appoint- 
ments (both scheduled and attended), academic plan completion, enrollment in spring 
2015, and student demographic information. 

Data for the outcome measures came from electronic records collected by the college’s 
counseling department and transcript data maintained by the district. 

The appointment scheduling data are based on all appointments scheduled by study stu- 
dents after random assignment (September 17, 2014) and before the end of the follow-up 
period (February 2, 2015). Data on attending appointments are based on all appointments 
scheduled between the beginning of random assignment (September 17, 2014) and the end 
of the follow-up period (February 2, 2015), as recorded in Sherpa. Some of these appoint- 
ments occurred after the end of the intervention period. 

Completion data are based on the number of students who did not have a counselor-ap- 
proved academic plan prior to random assignment (September 17, 2014) and completed an 
academic plan at least once before the end of the follow-up period (February 2, 2015), as 
recorded in Sherpa. It is possible for students to revise their academic plan over the course 
of the semester. Given the intervention’s goal of reducing the number of students who fail 
to ever complete an academic plan, analysis of this outcome measure did not take into 
account rates or patterns of revisions to academic plans. The completion outcome, there- 
fore, represents whether student participants completed an academic plan at any point 
during the data collection period. 

Academic persistence was defined as registering for at least one class in the spring semester 
of 2015 by February 27, 2015. This included both credit-bearing and non-credit-bearing 
classes. 
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Qualitative data collection, processing, and analysis for the implementation study 

The research team intended to conduct three focus groups (one for each research group) of 
8-10 students each, a focus group of 8-10 counselors, interviews with three administrators, 
and an interview with two support staff. Researchers also interviewed some students who 
were unable to attend the scheduled workshop and one-on-one focus groups. In the end, 
researchers were able to collect data from five workshop students (in one focus group and 
two individual interviews), three one-on-one students (in one focus group and one inter- 
view), one control student, eight counselors (in one focus group), three administrators, and 
two support staff (table Bl). 

Each of these activities was audio recorded, and each audio recording was later transcribed. 
Transcriptions were coded and analyzed according to a codebook covering topics such as 
the content of each session and students’ attitudes and perceptions of their session. Coding 
took place in Dedoose, an online qualitative data analysis software. 

Researchers also observed three one-on-one counseling sessions and two workshops. Notes 
from these activities were analyzed for patterns across sessions in each study group. 

Criteria for participation in the study 

To participate in the study, students had to meet the following criteria: 

• Were first-time freshmen at Saddleback College. 

• Had not completed an approved academic plan. 

• Were enrolled in at least one credit-bearing class. 

• Were not enrolled in special programs or did not have special status involving a 
mandatory or dedicated counseling component, such as those for student athletes 
or international students. 

• Were not enrolled in counseling courses that include academic plan completion. 

The college decided to target first-time students because members of this cohort would 
be the first to confront the registration block if they had completed 15 credits but did not 
have an academic plan by the following spring. This decision ruled out including con- 
tinuing students in the study group, who, according to the college, had advising needs 


Table Bl. Qualitative data collection activities and number of participants, 2014/15 

1 Activity 

Number I 

Administrator interviews 

3 

Counselor focus group 

8 

Support staff interviews 

2 

Workshop focus group 

5 

Workshop interviews 

2 

One-on-one focus group 

2 

One-on-one interviews 

1 

Control interview 

1 

Source: Authors' records of 2014/15 data collection activities led by the South Orange County Community 
College District. 
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that could have been more complex than those of freshmen because they would have had 
transcripts from the college or from other colleges to account for in their academic plans. 

For descriptions of the characteristics of the baseline and analytic samples of students, see 
tables Cl and C2 in appendix C. 

Random assignment of students to the interventions and control condition 

Randomization of the 1,839 students who met the initial eligibility criteria occurred in the 
third week of September 2014- A spreadsheet with the student IDs of these students was 
provided to Regional Educational Laboratory West for random assignment. The numbers 
of students assigned to the workshop and one-on-one groups were informed by two factors: 
the capacity of the college’s counseling department to dedicate counselors to delivering 
MAP services, and requirements in the research design to have sufficient power to detect 
minimal detectable effects. The college decided that offering 6 workshops and 20 one- 
on-one sessions per week for 10 weeks was optimal, given the number of counselors who 
were trained and available to lead these sessions. As a result, 1,140 students were initially 
assigned to the workshop group, 200 students were assigned to the one-on-one group, and 
499 students were assigned to the control group. 

After assignment, the study team provided the district with the assignment information for 
all 1,839 students for entry into the district’s data system. Email messages were sent to all stu- 
dents in the study, including general information about their enrollment in the study, instruc- 
tions explaining how to opt out of the study if desired, and basic information about the MAP 
tool. A total of 24 students opted out of the study shortly after random assignment, mostly 
because they had already completed their academic plan and were therefore not eligible for 
the intervention. An additional 52 students were identified as ineligible during the analy- 
sis because they had completed an academic plan prior to the date of random assignment. 
These students were removed from the analysis sample, leaving a total of 1,763 students. 

Outcome measures 

This study had four outcome measures: 

• Scheduling a counseling appointment. 

• Attending a counseling appointment. 

• Completing an academic plan. 

• Reenrolling in the semester following the intervention (academic persistence). 

Analytic approach 

There are two general classes of research: confirmatory and exploratory. Confirmatory 
research is designed to answer substantive empirical questions with a high level of confi- 
dence. To accomplish this, a confirmatory research question is clearly specified before data 
are collected and used to guide the analysis of study results. Confirmatory questions are 
usually designed to assess the validity of a program model or theory and identify which 
outcomes should be analyzed. The primary focus is to avoid erroneously concluding that 
there is a relationship when, in fact, one does not exist. Avoiding such errors requires a 
large enough sample of participants to ensure that it is possible to detect program effects 
that have practical significance. 


B-3 


Figure Bl. Study intake chart of sampled students, 2014/15 



Source: Authors' compilation. 


The goal of exploratory research is to identify relationships or patterns of relationships that 
can form the basis for future research. Exploratory research questions are not answered 
with the same level of confidence as confirmatory questions and are sometimes defined 
after a study has been designed and the data collected. As a result, the study design and 
number of participants are generally not sufficient to provide definitive answers about 
whether there is a “true” relationship between the variables being examined. The results of 
exploratory analyses are therefore considered suggestive. 

All impact estimates comparing the workshop group or the one-on-one group to the 
control group (standard services) were calculated using the following model: 

r = « + P,T„ + [i 2 T 2l + e, 

where T is the outcome of interest (such as academic plan completion) for student i; a is the 
average outcome for students assigned to the control group; is the estimated impact of 
the one-on-one condition compared to the control condition; (J 2 is the estimated impact of 
the workshop condition compared to the control condition; T u indicates whether student 
i was assigned to the one-on-one group; T 2j indicates whether student i was assigned to the 
workshop group; and e. is an error term. 
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Because of random assignment, ^ and (2 2 are unbiased estimates of the intent-to-treat 
effects for each of the two intervention groups, relative to the counterfactual of the control 
group. The estimated impact of the workshop condition compared to that of the one-on- 
one condition, and its standard errors, were calculated from the estimates associated with 

Pi and h 

All impact estimates comparing the workshop group to the one-on-one group were calcu- 
lated using the following model: 


Y. - a + S.T.. + g. 

I “1 ll I 

where Y is an outcome of interest (such as completion of an academic plan) for student i; 
a is the average outcome for students assigned to the one-on-one group; (2j is the estimated 
impact of the workshop condition compared to the one-on-one condition; T u indicates 
whether student i was assigned to the workshop group; and £. is an error term. The stan- 
dard errors for the estimated impact of the workshop condition compared to the one-on- 
one condition were calculated from the estimate associated with (3 r 

Before the study power analyses were conducted to estimate the minimal detectable effect 
for the difference between the workshop group and the one-on-one group for the academic 
plan completion outcome. The minimal detectable effect is the smallest true impact that 
an experiment is likely to detect (Bloom, 1995). The smaller the minimal detectable effect, 
the more likely it is that a study will be able to detect impacts of a small magnitude. The 
minimal detectable effect ( MDE ) for a binary outcome was calculated using the following 
formula (from Bloom, 1995): 


MDE = 2.80 * 


M 


7t( 1 - 7r)(l - r 2 ) 

T( 1 - T)n 


where 2.80 is the appropriate multiplier for 80 percent power and a 5 percent significance 
level with a two-tailed hypothesis test; n is the proportion of the study population that 
would have a value of 1 for the binary outcome in the absence of the intervention; r 2 is the 
explanatory power of the impact regression (assumed to be 0); T is the proportion of the 
sample that is randomly assigned to the intervention group; and n is the total size of the 
study sample. 

The minimal detectable effects for the two confirmatory research questions were calcu- 
lated to compare assignment to the workshop group and assignment to the control group 
and to compare assignment to the one-on-one group with assignment to the control group. 
The district provided information to establish the baseline value for the primary outcome 
of interest. The district estimated that about 30 percent of students enrolled at Saddleback 
College in the fall of 2013 had at least started an academic plan, but only about 10 percent 
had completed an academic plan that had been reviewed and approved by a counselor. 
The power calculations assumed a 5 percent significance level, 80 percent power, a two- 
tailed test, and a baseline completion rate of 20 percent. 

In comparisons of outcomes for students assigned to the workshop group with outcomes for 
those assigned to the control, the total sample size for the power analysis was estimated to 
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be 1,639 (1,140 students in the workshop group and 499 students in the control group). The 
relative probability of assignment to the workshop group was 0.70. The minimal detectable 
effect is 6.0 percentage points. An equivalent calculation for the one-on-one group yields a 
minimal detectable effect of 9.3 percentage points. The sample size for that calculation was 
699 students, with a probability of assignment to the one-on-one group of 0.29. 

Minimal detectable effects can also be calculated for analyses that compare outcomes for 
students assigned to the workshop group to outcomes for those assigned to the one-on-one 
group. The total sample size for that comparison was 1,340 (1,140 students in the work- 
shop group and 200 students in the one-on-one group), and students were assigned to the 
workshop group with a relative probability of 0.85. This power analysis showed that with 
a completion rate for the one-on-one group of 20 percent, the minimal detectable effect is 
8.6 percentage points. 

Statistical adjustments 

The number of confirmatory research questions for this study was limited to two: 

• Does assignment to the workshop group increase the academic plan completion 
rate, compared with the control group? 

• Does assignment to the one-on-one group increase the academic plan completion 
rate, compared with the control group? 

Because these two questions fall within the same domain and are not independent of each 
other, the Benjamini and Hochberg (1995) method was used to adjust for multiple compar- 
isons when testing these two hypotheses. This approach uses a stepwise method to adjust 
obtained p-values, based on the number of outcome variables within the domain and the 
number of hypothesis tests being conducted. The adjusted and unadjusted results are pre- 
sented in table B2. 

Because individual students (rather than groups of students) were randomly assigned, 
the main analysis did not adjust for clustering. There is disagreement over whether such 
adjustment is necessary with individual randomization when the intervention is delivered 
in groups (see, for example, Serlin, Wampold, & Levin, 2003; Siemer & Joorman, 2003a, 
2003b; Weiss, Lockwood, & McCaffrey, 2014). Moreover, the data needed to associate stu- 
dents with unique workshops were not available, so analyses that account for clustering 
in the data could not be performed. Such analyses, however, are unlikely to change the 


Table B2. Multiple hypothesis adjustment for confirmatory outcomes, 2014/15 



Workshop group 

One-on-one group 

Control group 

Difference between 
workshop and control groups 

Difference between 
one-on-one and control groups 

Outcome 

measure 

Standard 
Percent deviation 

Standard 
Percent deviation 

Standard 
Percent deviation 

Percent- 
age point Raw Adjusted 

difference p-value p-value 

Percent- 
age point Raw Adjusted 

difference p-value p-value 


Completed an 

academic plan 38.2 48.6 42.0 49.5 18.8 39.1 19.5*** <.001 <.001 23.2*** <.001 <.001 


*** Significant at p < .001. 

Note: Pre-intervention, there were 1,763 students in the full analytic sample: 1,085 students in the workshop group, 193 students in 
the one-on-one group, and 485 students in the control group. Rounding may cause slight discrepancies in differences, p-values were 
adjusted for multiple comparisons using the Benjamini-Hochberg (1995) method. 

Source: Authors' calculations using 2015 administrative data from the South Orange County Community College District. 
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substantive conclusions, given the large size of the estimates reported and the very small 
standard errors associated with those estimates. Because counselors were not randomly 
assigned, it is possible that the impact estimates may be affected by the effectiveness of 
individual counselors, rather than by the intervention itself. This also appears unlikely, 
however, because the estimates for attending an appointment are nearly identical to the 
estimates for completing an academic plan, suggesting that nudging students to attend 
appointments or offering guaranteed access to appointments was the main driver of the 
impacts. 

Missing data 

Students who did not have a data record for an outcome such as appointment scheduh 
ing or academic plan completion were coded as not having completed that outcome. For 
example, if a student was not included in the list of students who had created an academic 
plan, the completion data for that student was not considered to be missing; instead, the 
lack of data was taken as indicating that an academic plan had not been created. Similarly, 
students were coded as having zero academic counseling appointments if no advisement 
records for the student were included in the data. 

The data for counseling appointment attendance included 36 students with data records 
but no values recorded for attendance. Attendance for those students was coded as “no” 
to indicate that they did not attend the scheduled appointment. Sensitivity analyses were 
conducted, coding the missing outcomes to missing, yes, and no, to confirm that the deci- 
sion did not affect the findings for that outcome. These analyses are included in table B3. 


Table B3. Sensitivity analysis for missing attendance values, 2014/15 


Missing attendance code 

Workshop 

group 

(percent) 

One-on-one 

group 

(percent) 

Control 

group 

(percent) 

Difference 
between 
workshop and 
control groups 
(percentage 
points) 

Difference 
between 
one-on-one and 
control groups 
(percentage 
points) 

Difference 
between 
workshop and 
one-on-one 

groups 

(percentage 

points) 

Attendance = no 

48.2 

50.3 

30.1 

18.1*** 

20 2*** 

-2.1 

Attendance = yes 

50.4 

51.3 

32.2 

18.2*** 

19.1*** 

-0.9 

Sample size (n = 1,763) 

1,085 

193 

485 

na 

na 

na 

Attendance code is missing 

49.3 

50.8 

30.7 

18.6*** 

20 0*** 

-1.5 

Sample size (n = 1,727) 

1,061 

191 

475 

na 

na 

na 


*** Significant at p < .001. 
na is not applicable. 

Note: Rounding may cause slight discrepancies in differences. All statistical tests were conducted using the ordinary least squares 
regression models described in the analytic approach section of appendix B. 

Source: Authors' calculations using 2015 administrative data from the South Orange County Community College District. 
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Appendix C. Supplemental tables 


Pre-intervention sample sizes and characteristics for the baseline and analytic samples are 
in tables Cl and C2. Postintervention outcomes for the analytic sample and the estimated 
effects are in table C3. Student responses to a survey on the MAP workshop are in table 
C4 and figures Cl, C2, and C3. Detailed data on cost per student for each counseling 
approach are in table C5. 


Table Cl. Pre-intervention sample sizes and characteristics for the baseline sample, 2014/15 



Full 

Workshop 

One-on-one 

Control 

Difference between 
workshop and 
control groups 

Difference between 
one-on-one and 
control groups 

Difference between 
workshop and 
one-on-one groups 

Characteristic 

sample 

(percent) 

group 

(percent) 

group 

(percent) 

group 

(percent) 

Percent- 
age points 

p-value 

Percent- 
age points 

p-value 

Percent- 
age points 

p-value 

Gender 

Male 

51.2 

49.8 

49.2 

54.9 

-5.1 

0.06 

-5.7 

0.18 

0.6 

0.88 

Age 

Younger than 19 years old 

70.1 

70.7 

68.5 

69.3 

1.4 

0.58 

-0.8 

0.83 

2.2 

0.53 

Average age (years) 

21.1 

20.9 

21.9 

21.3 

-0.4 

0.41 

0.6 

0.50 

-1.0 

0.19 

Race/ethnicity 

Asian or Pacific Islander 

6.5 

7.5 

4.1 

5.1 

2.4 

0.08 

-1.0 

0.63 

3.4 

0.08 

Black 

1.2 

1.6 

0.5 

0.6 

0.9 

0.11 

-0.1 

0.91 

1.0 

0.22 

Hispanic 

12.8 

12.8 

10.8 

13.8 

-1.0 

0.58 

-3.0 

0.30 

2.0 

0.46 

White 

56.7 

57.0 

61.3 

54.1 

2.9 

0.29 

7.2 

0.09 

-4.4 

0.26 

Other race/multiracial 

22.8 

21.1 

23.2 

26.3 

-5.2** 

0.02 

-3.1 

0.38 

-2.1 

0.53 

Enrollment status in fall 2014 

Full-time 

48.9 

49.1 

45.0 

49.9 

-0.8 

0.77 

-4.9 

0.24 

4.1 

0.29 

Part-time 

42.1 

42.1 

43.0 

41.9 

0.2 

0.93 

1.1 

0.79 

-0.9 

0.82 

New to South Orange County 
Community College District 

82.9 

84.5 

80.5 

80.4 

4.1** 

0.04 

0.1 

0.97 

4.0 

0.17 

Enrolled in noncredit courses 

5.2 

5.4 

5.5 

4.6 

0.8 

0.52 

0.9 

0.63 

-0.1 

0.94 

Not enrolled 3 

3.8 

3.4 

6.5 

3.6 

-0.2 

0.84 

2.9 

0.07 

-3.1 

0.04 

Main reason for enrolling in college b 

Complete a certificate program 

2.1 

1.9 

0.5 

3.4 

-1.5* 

0.05 

—2.9** 

0.02 

1.4 

0.21 

Obtain an associate’s degree 

55.5 

55.4 

59.0 

54.3 

1.1 

0.69 

4.7 

0.26 

-3.6 

0.34 

Transfer to a four-year 
college/university 

66.4 

64.9 

69.5 

68.7 

-3.9 

0.13 

0.8 

0.85 

-4.6 

0.20 

Obtain/update job skills 

5.3 

5.8 

4.5 

4.6 

1.2 

0.32 

-0.1 

0.95 

1.3 

0.44 

Other 

20.6 

21.1 

19.5 

19.6 

1.5 

0.49 

-0.1 

0.97 

1.6 

0.60 

Diplomas/degrees earned 

High school diploma 

84.8 

85.8 

83.0 

83.2 

2.7 

0.17 

-0.2 

0.96 

2.8 

0.30 

General Educational 











Development certificate 

3.7 

3.0 

5.0 

4.8 

-1.9 

0.07 

0.2 

0.90 

-2.0 

0.16 

Other secondary credential 

8.0 

8.1 

8.5 

7.8 

0.2 

0.87 

0.7 

0.76 

-0.4 

0.84 

None of the above 

3.5 

3.1 

3.5 

4.2 

-1.1 

0.28 

-0.7 

0.64 

-0.4 

0.80 

Date of high school graduation/receipt of General Educational Development certificate 

During the past year 

83.0 

83.8 

80.0 

82.5 

1.3 

0.52 

-2.5 

0.44 

3.8 

0.20 

1-5 years ago 

6.9 

6.5 

6.8 

7.6 

-1.0 

0.45 

-0.8 

0.73 

-0.3 

0.88 

6-10 years ago 

2.9 

2.9 

3.7 

2.5 

0.4 

0.69 

1.2 

0.42 

-0.8 

0.55 

More than 10 years ago 

7.2 

6.7 

9.5 

7.4 

-0.6 

0.65 

2.1 

0.35 

-2.7 

0.18 


* Significant at p < .05; ** significant at p < .01. 

Note: Pre-intervention, there were 1,815 students in the full baseline sample: 1,116 students in the workshop group, 200 students in 
the one-on-one group, and 499 students in the control group. Rounding may cause slight discrepancies in differences. Missing values 
are included only in variable distributions for characteristics with more than 5 percent of the sample missing. 

a. Not-enrolled students withdrew from all courses in which they enrolled in fall 2014 prior to the census date. 

b. Responses are not mutually exclusive. 

Source: Authors' calculations using administrative data from the South Orange County Community College District, 2015. 
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Table C2. Pre-intervention student characteristics for the analytic sample, 2014/15 


Characteristic 

Full 

sample 

(percent) 

Workshop One-on-one 
group group 

(percent) (percent) 

Control 

group 

(percent) 

Difference between 
workshop and 
control groups 

Percent- 
age points p-value 

Difference between 
one-on-one and 
control groups 

Percent- 
age points p-value 

Difference between 
workshop and 
one-on-one groups 

Percent- 
age points p-value 

Gender 

Male 

51.2 

50.0 

48.4 

54.9 

-4.9 

0.08 

-6.5 

0.13 

1.6 

0.68 

Age 

Younger than 19 years old 

69.9 

70.5 

68.4 

69.3 

1.2 

0.62 

-0.9 

0.82 

2.1 

0.56 

Average age (years) 

21.1 

20.9 

21.9 

21.4 

-0.5 

0.35 

0.5 

0.52 

-1.0 

0.17 

Race/ethnicity 

Asian or Pacific Islander 

6.5 

7.6 

4.3 

5.1 

2.5 

0.07 

-0.8 

0.71 

3.3 

0.09 

Black 

1.2 

1.6 

0.5 

0.6 

1.0 

0.11 

-0.1 

0.92 

1.1 

0.22 

Hispanic 

12.7 

12.8 

10.2 

13.6 

-0.8 

0.67 

-3.4 

0.24 

2.6 

0.32 

White 

56.8 

57.0 

62.0 

54.4 

2.5 

0.36 

7.6 

0.08 

-5.1 

0.20 

Other race/multiracial 

22.7 

21.1 

23.0 

26.3 

-5.2** 

0.03 

-3.3 

0.37 

-1.9 

0.57 


Enrollment status in fall 2014 


Full-time 

48.6 

48.6 

45.1 

50.1 

-1.5 

0.58 

-5.0 

0.24 

3.5 

0.37 

Part-time 

42.2 

42.4 

42.5 

41.6 

0.7 

0.78 

0.8 

0.84 

-0.1 

0.98 

New to South Orange County 











Community College District 

5.3 

5.5 

5.7 

4.7 

0.8 

0.52 

1.0 

0.62 

-0.2 

0.92 

Enrolled in noncredit courses 

3.9 

3.5 

6.7 

3.5 

0.0 

1.00 

3.2 

0.05 

-3.2 

0.03 

Not enrolled 3 

83.4 

85.1 

80.8 

80.8 

4 2** 

0.04 

0.0 

1.00 

4.2 

0.14 

Main reason for enrolling in college b 

Complete a certificate program 

2.0 

1.8 

0.5 

3.3 

-1.5* 

0.05 

-2.8 

0.02 

1.2 

0.26 

Obtain an associate’s degree 

55.1 

55.0 

58.0 

54.0 

1.0 

0.71 

4.0 

0.34 

-3.0 

0.44 

Transfer to a four-year 











college/university 

66.0 

64.5 

68.9 

68.2 

-3.7 

0.15 

0.7 

0.87 

-4.4 

0.24 

Obtain/update job skills 

5.5 

6.0 

4.7 

4.7 

1.2 

0.32 

-0.1 

0.97 

1.3 

0.46 

Other 

21.0 

21.7 

20.2 

20.0 

1.7 

0.46 

0.2 

0.95 

1.5 

0.65 

Diplomas/degrees earned 

High school diploma 

84.6 

85.5 

82.9 

83.1 

2.4 

0.22 

-0.2 

0.95 

2.6 

0.35 

General Educational 











Development certificate 

3.8 

3.0 

5.2 

4.9 

-1.9 

0.07 

0.2 

0.89 

-2.1 

0.15 

Other secondary credential 

8.2 

8.2 

8.3 

8.0 

0.2 

0.91 

0.2 

0.92 

-0.1 

0.97 

None of the above 

3.5 

3.2 

3.6 

3.9 

-0.7 

0.49 

-0.3 

0.85 

-0.4 

0.78 

Date of high school graduation/receipt of General Educational Development certificate 

During the past year 

82.9 

83.6 

80.3 

82.3 

1.4 

0.52 

-1.9 

0.56 

3.3 

0.28 

1-5 years ago 

7.0 

6.6 

7.1 

7.6 

-0.9 

0.51 

-0.5 

0.83 

-0.5 

0.82 

6-10 years ago 

2.9 

3.0 

2.7 

2.6 

0.4 

0.68 

0.1 

0.93 

0.3 

0.85 

More than 10 years ago 

7.3 

6.7 

9.8 

7.6 

-0.8 

0.57 

2.3 

0.32 

-3.1 

0.14 


* Significant at p < .05; ** significant at p < .01. 

Note: Pre-intervention, there were 1,763 students in the full analytic sample: 1,085 students in the workshop group, 193 students 
in the one-on-one group, and 485 in the control group. Rounding may cause slight discrepancies in differences. Missing values are 
included only in variable distributions for characteristics with more than 5 percent of the sample missing. 

a- Not-enrolled students withdrew from all courses in which they enrolled in fall 2014 prior to the census date, 
b. Responses are not mutually exclusive. 

Source: Authors' calculations using 2015 administrative data from the South Orange County Community College District. 
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Table C3. Postintervention outcomes for the analytic sample and estimated effects, 2015 



Workshop group 

One-on-one group 

Control group 

Difference 
between 
workshop and 
control groups 

Difference 
between 
one-on-one and 
control groups 

Difference 
between workshop 
and one-on-one 
groups 

Outcome measure 

Percent 

Standard 

deviation 

Percent 

Standard 

deviation 

Percent 

Standard 

deviation 

Percent- 

age 

points 

p-value 

Percent- 

age 

points 

p-value 

Percent- 

age 

points 

p-value 

Student scheduled 













counseling appointment 

56.4 

49.6 

58.5 

49.4 

32.6 

46.9 

23 8*** 

<.001 

26 0*** 

<.001 

-2.1 

0.6 

Student attended 













counseling appointment 

48.2 

50.0 

50.3 

50.1 

30.1 

45.9 

18.1*** 

<.001 

20 2*** 

<.001 

-2.1 

0.6 

Student completed an 
academic plan 

38.2 

48.6 

42.0 

49.5 

18.8 

39.1 

19.5*** 

<.001 

23 2 *** 

<.001 

-3.7 

0.3 

Student enrolled in 













spring 2015 

77.8 

41.6 

77.2 

42.1 

79.8 

40.2 

-2.0 

0.4 

-2.6 

0.5 

0.6 

0.9 


*** Significant at p < .001. 

Note: Pre-intervention, there were 1,763 students in the full analytic sample: 1,085 students in the workshop group, 193 students in 
the one-on-one group, and 485 in the control group. Rounding may cause slight discrepancies in differences. All statistical tests were 
conducted using ordinary least squares regression models. See appendix B for more details. 

Source: Authors’ calculations using 2015 administrative data from the South Orange County Community College District. 


Table C4. Student responses to workshop survey, 2015 (percent) 


Statement 

Strongly 

agree 

Agree 

Neither 
agree nor 
disagree 

Disagree 

Strongly 

disagree 

Did not 

answer 

Total 

number of 
respondents 

This workshop met my 
academic planning needs. 

47.8 

36.2 

13.5 

0.6 

1.2 

0.6 

163 

My plan is to use this 

MAP in order to reach my 
education goal while at 
Saddleback College. 

57.6 

31.3 

8.6 

0.6 

0.6 

1.2 

163 

1 was satisfied with the 

workshop. 

55.8 

30.1 

9.2 

3.1 

1.2 

0.6 

163 


MAP is an academic plan created using the My Academic Plan tool. 

Note: Results from this survey should be viewed with caution. The survey data may include responses from some nonstudy students 
who had previously attended a MAP group session that was not associated with the study (although, according to the counselors, non- 
study group sessions were conducted in the same manner as the study workshops). Rounding may cause slight discrepancies in sums 
and differences. 

Source: Authors' calculations using 2015 administrative data from the South Orange County Community College District. 
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Figure Cl. Student responses to workshop survey (What did you like/find helpful 
about the workshop?), 2014/15 


Percent 



The advice and The assistance The assistance The Other 

guidance from in choosing the in planning of one-on-one 

counselors correct classes future goals help 


Note: n = 151. Results from this survey should be viewed with caution. According to counseling staff, the sur- 
vey data may include responses from some nonstudy students who had previously attended a My Academic 
Plan group session that was not associated with the study (although, according to the counselors, nonstudy 
group sessions were conducted in the same manner as the study workshops). Response categories were 
derived from an analysis of open-ended survey responses. 

Source: Authors' calculations using 2015 administrative data from Saddleback College. 


Figure C2. Student responses to workshop survey (What did you not like/not find 
helpful about the workshop?), 2014/15 


Percent 
50 -i 



Nothing The workshop There were The software There was Other 

was too long not enough had limitations a lack of 

counselors one-on-one time 


Note: n = 151. Results from this survey should be viewed with caution. The survey data may include re- 
sponses from some nonstudy students who had previously attended a My Academic Plan group session that 
was not associated with the study (although, according to the counselors, nonstudy group sessions were 
conducted in the same manner as the study workshops). Response categories were derived from an analysis 
of open-ended survey responses. 

Source: Authors' calculations using 2015 administrative data from Saddleback College. 
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Figure C3. Student responses to workshop survey (How could we improve the 
workshop?), 2015 


Percent 
50 -i 



Nothing 

More 

More 

Fewer 

Expand 

Separate 

Other 


counselors/ 

one-on-one 

students in 

software's 

students 



aides 

time 

the workshops 

ability 

according to 
major, goal, etc. 



Note: n = 151. Results from this survey should be viewed with caution. The survey data may include re- 
sponses from some nonstudy students who had previously attended a My Academic Plan group session that 
was not associated with the study (although, according to the counselors, nonstudy group sessions were 
conducted in the same manner as the study workshops). Response categories were derived from an analysis 
of open-ended survey responses. 

Source: Authors' calculations using 2015 administrative data from Saddleback College. 


Table C5. Average per student cost by component for each counseling approach, 2014/15 ($) 


1 Component 

Workshop group 

One-on-one group 

Control group 1 

Cost per student 

27 

46 

24 

Counselor time in session 

11 

35 

20 

Counselor time in training 

0 

2 

1 

Computer lab 

9 

0 

0 

Administrative staff time to support training 

0 a 

2 

1 

Nudges (emails and texts) 

1 

1 

0 

Robocalls 

1 

1 

0 

Administrative staff time to support scheduling of sessions 

4 

5 

3 

Alternative cost per study student (with all full-time staff) b 

34 

53 

28 

Sample size (number) 

1,085 

193 

485 


Note: Costs are driven by the number of counseling sessions scheduled by students in the respective study groups. If a student fails 
to attend a scheduled session, the analysis assumes that the cost associated with the session is still realized. The average hourly rate 
for a typical counselor in each group is based on the fraction of counselors that were full-time compared to part-time. The cost of com- 
puters needed for the workshops is based on 19 computers at $500 each. The cost of the Sherpa tool is considered part of the base 
college services. Therefore it is not associated with an incremental cost across the three groups. If another college was considering 
this intervention but lacked such a counseling support system the cost of such a system would need to be considered. 

a. Unrounded value is 0.31. 

b. The sensitivity adjustment is included to communicate the role different compensation rates associated with full-time and adjunct 
counselors play in the cost of each modality. The alternative describes the cost assuming all counselors are full-time staff and that full- 
time counselors cost 74 percent more than adjuncts. Specifically, it assumes the annual cost of salary and benefits for a full-time coun- 
selor is $139,200 ($120,000 salary plus 16 percent for benefits) compared with $80,000 for adjuncts ($80,000 salary and no benefits). 

Source: Cost estimates are based on study team calculations from program expenditure data for the intervention semesters (fall 2014 
and spring 2015) as reported to Regional Educational Laboratory West by Saddleback College and research on full-time and adjunct 
counselor salaries as reported by the California State Controller’s Office for the same time period. 
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Appendix D. Descriptions of MySite, Sherpa, and My Academic Plan systems 


This appendix includes descriptions of MySite, Sherpa, and the My Academic Plan (MAP) 
system. 7 

MySite 

MySite is a web portal used by students, faculty, staff, and administrators within the South 
Orange County Community College District. It was developed in-house by the district’s 
technical staff and delivers all of the districtwide online services used by Saddleback 
College and Irvine Valley College. 

Sherpa 

Sherpa is a proactive and interactive recommendation engine developed entirely by dis- 
trict technical staff to enhance student success. The Sherpa system generates intelligent, 
automated computer recommendations (similar to those used by companies such as eBay, 
Netflix, Amazon, Pandora, and Apple) to assist students in making well-informed decisions 
and engaging in behaviors that are likely to improve their chances of success. An image 
of the Sherpa dashboard is shown in figure Dl. Sherpa delivers “just-in-time” information, 
using time-, event-, or data-based triggers to deliver reminders and recommendations — 
called “nudges” — through a feed on the student web portal (MySite), email, text messag- 
es, text-to-speech audio, or a mobile phone application. Images of nudges on the Sherpa 
website are shown in figures D2 and D3. Nudges can encourage students to find acceptable 
alternatives when their preferred courses are full, target at-risk students for academic inter- 
ventions, tailor information about campus events to individual interests and needs, and 
(for this study) encourage students to complete an academic plan. Fully integrated with the 


Figure Dl. Sherpa dashboard 
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Note: Used by authorized college staff to create profiles and nudges. 

Source: Saddleback College website, https://mysite.socccd.edu/Portal/MySite/Sherpa/Dashboard.aspx. 
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Figure D2. MySite To-Do List item (or nudge) 


MySite 


B MvSite 

□ My Classes 
0 My Information 
0 My Academic Plan 
III My Email 

> Change PIN/Password 

► My College 

> Contact Us 

* My Preferences 
» Parking 

► Help 


TODAY IS: 

JUN 28 


^Welcome 


To-Do List (0 new) 


O Add Task 


Expand All Collapse All 


Done Priority Task Deadline * 

O High S Sign up for MAP Advisement Jul 16, 2016 
Session 

Congratulations on creating your First 
Semester Plan - that's a great step toward 
your success. Now ... fmore' 


Setting; 

< > 


Today 


Jun 19 - 25 


Show completed tasks 


Sh91.' ftl kill 
M W D 


Sun 

Mon 

Tue 

Wed 

Thu 

Fri 

Sat 

6/19 

6/20 

6/21 

6/22 

6/23 

6/24 

6/25 

all-day 








8:00 am 
Registra 
Appointi 
• Fall 
2016 


News Feed (0 new) 


Start Your lob Search with Gaucho 

a May 27. 2016 12:07 PM - General - Saddlet 

Gaucho Jobs is your official employment 
Search job and internship postings (1000 
resume create a portfolio, join Resume f 

... f morel 

i ; H Library Summer Workshops 
S Jun 20. 2016 9:21 AM - General - IVC 

Need some assistance with researc 
by attending a Library Summer Wo 
topics dedicated to helping you find 

... f morel 


| Registration for Fall 2016 

a Apr 29, 2016 10:12 AM - General - IVC 

Fall 2016 registration begins May 2. Log 
registration time. From the left side me 
information then Appointments.... fmor 


El Aft 


Source: Saddleback College website, https://mysite.socccd.edu/Portal/MySite/Default.aspx. 


Figure D3. MySite To-Do List item (or nudge) with detail 
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Source: Saddleback College website, https://mysite.socccd.edu/Portal/MySite/Default.aspx. 
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district’s student information system, the district’s home-built student records management 
system, MySite, and MAP, Sherpa makes these personalized recommendations through 
the use of profiles that describe sets of students with common characteristics. 

My Academic Plan 

MAP is among the many tools that students see when they log on to their personal MySite. 
An interactive program that helps students select an education goal, choose a major, and 
identify the courses that they need to satisfy the requirements of that major, MAP is tightly 
integrated with MySite, Sherpa, the online class schedule, and Project ASSIST (Articula- 
tion System Stimulating Interinstitutional Student Transfer), a statewide articulation data- 
base that includes every transferable course offered by every community college in the state. 

When students log on to MySite and select their intended education goals and programs 
of study or their major, they immediately see a list of both required courses and elective 
options. The list is also personalized. For example, if a student’s placement test score 
requires him or her to take a developmental education course in mathematics or reading, 
that course will pre-populate the schedule. Students can select and add courses from a 
list of courses designated as general education requirements, major/program requirements, 
electives, prerequisites/corequisites, or courses that are not applicable to the selected edit- 
cation goal. Students’ choices are then automatically arrayed in a semester-by-semester 
plan, leading to the desired credential in the amount of time specified by the student. 

The MAP system was developed by the information technology team at the district and 
was launched in 2007 after soliciting input from counselors and students. MAP is similar 
enough to other academic planning tools that the results of this study would be applicable 
in other settings (see figure D4). 


Figure D4. Desktop view of My Academic Plan tool 
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Source: Saddleback College website, https://mysite.socccd.edu/Portal/IVIAP/Wizard/CourseSelection.aspx. 
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Notes 


1. The median was 5 and there were two modes: 2 and 8. 

2. Results from this survey should be interpreted with caution because the survey was 
not consistently offered and because it includes responses from up to five students who 
were not included in the study but attended a workshop session that was not associated 
with the study (although, according to counselors, nonstudy workshop sessions were 
conducted in the same manner as the study workshops). For these reasons, a response 
rate calculation is included. 

3. Counselors’ opinions and concerns were observed directly by the research team during 
focus groups and interviews (see appendix B). 

4. The study was designed to detect differences of about 8.6 percentage points or larger 
between the workshop and the one-on-one groups. If the true difference is smaller, the 
study would be unable to detect it and differences could be undetected as a result. See 
discussion of statistical power in appendix B for details. 

5. To learn more about the online system used at Saddleback College, contact the dis- 
trict’s director of information technology (Academic Systems) at jgaston@socccd.edu 
or see https://www.socccd.edu/technologyandlearning/tl_projects_student.html. 

6. The design allowed for 1,140 workshop slots and 200 one-on-one session slots. 

7. More information on each of these tools can be found at https://www.socccd.edu/tech- 
nologyandlearning/tl_projects_student.html. 
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